2024-03-25T18:21:46.535 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:21:46.535 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/orig.config.yaml 2024-03-25T18:21:46.573 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:21:46.661 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621323' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi018.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIf8qtBW1VRnU77oStvxOR7mFLCKGb1HTnzzU0QgDPNLlMjYNsWy8DxR7LMFeOQOQ9GDGrEIsPPh/wjTS8d9ASk= smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4uEslQyHioVMal//nl/QuVFFOzb5Vxx/XC+yGG8DaL9Py/goUzcz2fw0wljTg06zHW5budGXLp7J9jX4RObhg= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:21:46.661 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:21:46.662 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:21:46.662 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:21:46.663 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:21:46.684 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:21:46.684 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:21:46.684 INFO:teuthology.packaging:ref: None 2024-03-25T18:21:46.684 INFO:teuthology.packaging:tag: None 2024-03-25T18:21:46.684 INFO:teuthology.packaging:branch: main 2024-03-25T18:21:46.684 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:21:46.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:21:46.998 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:21:46.999 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:21:47.009 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:21:47.009 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:21:47.047 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:21:47.061 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:21:47.072 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:21:47.097 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:10.922456', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIf8qtBW1VRnU77oStvxOR7mFLCKGb1HTnzzU0QgDPNLlMjYNsWy8DxR7LMFeOQOQ9GDGrEIsPPh/wjTS8d9ASk='} 2024-03-25T18:21:47.118 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:10.924560', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4uEslQyHioVMal//nl/QuVFFOzb5Vxx/XC+yGG8DaL9Py/goUzcz2fw0wljTg06zHW5budGXLp7J9jX4RObhg='} 2024-03-25T18:21:47.118 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:21:47.128 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:21:47.129 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:21:47.129 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:21:47.208 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f34a65f80d0>, signals=[15]) 2024-03-25T18:21:47.208 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:21:47.219 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:21:47.219 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T18:21:47.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:21:47.327 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T18:21:47.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:21:47.431 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:21:47.442 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2024-03-25T18:21:47.460 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2024-03-25T18:21:47.460 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2024-03-25T18:21:47.516 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="CentOS Stream" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="9" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:ID="centos" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="9" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:ANSI_COLOR="0;31" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:21:47.517 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:21:47.518 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2024-03-25T18:21:47.537 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-03-25T18:21:47.554 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-03-25T18:21:47.554 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:21:47.611 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:21:47.612 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-03-25T18:21:47.629 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:21:47.639 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:21:47.652 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:21:47.653 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:21:47.655 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:21:47.670 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:21:47.680 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:21:47.680 DEBUG:teuthology.orchestra.run.smithi018:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:21:47.711 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:21:47.737 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:21:47.795 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:21:47.813 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:21:47.813 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:21:47.813 DEBUG:teuthology.orchestra.run.smithi018:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:21:47.814 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:21:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:47.829 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-03-25T18:21:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:47.830 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2024-03-25T18:21:47.885 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:21:47.886 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-361.el9.x86_64 2024-03-25T18:21:47.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-03-25T18:21:47.887 INFO:teuthology.orchestra.run.smithi018.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:21:47.887 INFO:teuthology.task.kernel:Running kernel on smithi018: 5.14.0-361.el9.x86_64 2024-03-25T18:21:47.887 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum install -y kernel 2024-03-25T18:21:49.996 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:21:50.046 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T18:21:51.454 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-25T18:21:51.505 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-25T18:21:54.851 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:03 2024-03-25T18:21:55.384 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-25T18:21:59.704 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 19 MB 00:03 2024-03-25T18:22:00.245 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 19 MB 00:03 2024-03-25T18:22:06.874 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:22:08.295 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 16 kB 00:00 2024-03-25T18:22:08.425 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.1 MB 00:04 2024-03-25T18:22:10.041 INFO:teuthology.orchestra.run.smithi018.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T18:22:10.119 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-25T18:22:13.787 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 5.8 MB/s | 21 MB 00:03 2024-03-25T18:22:15.088 INFO:teuthology.orchestra.run.smithi018.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T18:22:17.319 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:22:17.320 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:22:17.409 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:22:17.420 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:22:17.420 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout:Install 4 Packages 2024-03-25T18:22:17.421 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:22:17.422 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 76 M 2024-03-25T18:22:17.422 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 123 M 2024-03-25T18:22:17.422 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:22:17.858 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:17.858 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:17.860 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:18.223 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:18.248 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:18.258 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:18.266 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:18.278 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:18.283 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:18.284 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:18.321 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:18.322 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:18.383 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:18.392 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:18.405 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:18.453 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:18.462 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:18.484 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:18.600 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:18.618 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:18.646 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:18.686 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:18.692 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:18.775 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:18.833 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:18.866 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:18.903 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:18.912 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T18:22:19.013 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:19.018 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:19.063 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:19.103 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:19.106 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:19.163 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:19.171 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:19.197 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:19.228 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:19.228 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:19.228 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:19.289 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:19.347 INFO:teuthology.orchestra.run.smithi018.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 40 kB/s | 66 kB 00:01 2024-03-25T18:22:19.730 INFO:teuthology.orchestra.run.smithi018.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 7.3 MB/s | 15 MB 00:02 2024-03-25T18:22:19.896 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:20.058 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:20.075 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:20.109 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:20.189 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:20.540 INFO:teuthology.orchestra.run.smithi018.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-25T18:22:20.541 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:20.576 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:20.709 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:20.854 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:22:20.887 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:20.946 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:21.169 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:22:21.169 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:22:21.269 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:22:21.276 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:22:21.276 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-03-25T18:22:21.277 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:22:21.277 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:22:21.277 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:22:21.277 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:22:21.277 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:22:21.277 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:22:21.278 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:22:21.278 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:22:21.278 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:22:21.278 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:22:21.278 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-03-25T18:22:21.279 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:22:21.280 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 76 M 2024-03-25T18:22:21.280 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 123 M 2024-03-25T18:22:21.280 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:22:21.438 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:21.440 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:21.440 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:21.578 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:21.588 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:21.611 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:21.622 INFO:teuthology.orchestra.run.smithi018.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T18:22:21.626 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:21.626 INFO:teuthology.orchestra.run.smithi018.stdout:Total 18 MB/s | 76 MB 00:04 2024-03-25T18:22:21.661 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:21.668 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:21.692 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:21.817 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:21.833 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:21.835 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:21.869 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:21.869 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:21.903 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:21.937 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:21.938 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:21.956 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:21.972 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:21.974 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:21.994 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:22:22.165 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:22.166 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:22.178 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:22.321 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:22:22.541 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:22.576 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:22.581 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:22.697 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 50 kB/s | 66 kB 00:01 2024-03-25T18:22:22.768 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:22:22.768 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:22:22.870 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:22.977 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:22:23.247 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 8.0 MB/s | 15 MB 00:01 2024-03-25T18:22:23.287 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:23.358 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:22:23.656 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 15 MB/s | 33 MB 00:02 2024-03-25T18:22:23.702 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:23.719 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:22:23.793 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:22:23.794 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:22:23.930 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:22:24.556 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 15 MB/s | 27 MB 00:01 2024-03-25T18:22:24.563 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:24.563 INFO:teuthology.orchestra.run.smithi053.stdout:Total 23 MB/s | 76 MB 00:03 2024-03-25T18:22:25.259 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:22:25.702 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:22:25.703 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:22:26.478 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:22:26.752 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:22:26.752 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:22:27.116 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:27.215 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:29.368 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:22:30.006 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:30.106 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:30.149 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:30.285 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:33.009 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:33.143 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:35.394 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:35.436 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:38.219 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:38.262 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:40.604 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:43.357 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:11.560 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:11.560 INFO:teuthology.orchestra.run.smithi018.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:11.560 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:23:11.607 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:11.933 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:11.933 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:11.933 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:11.933 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:12.389 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:12.389 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:23:12.389 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:23:12.389 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:12.389 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:12.389 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:12.390 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:12.390 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:23:12.390 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:23:12.726 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:12.841 INFO:teuthology.orchestra.run.smithi018.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:12.841 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:12.841 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:12.841 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:12.841 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:12.841 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:23:12.841 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi018.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:12.841 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:12.924 INFO:teuthology.orchestra.run.smithi018.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:12.924 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:12.924 DEBUG:teuthology.orchestra.run.smithi018:> sudo rpm -qi grub2-tools 2024-03-25T18:23:12.972 INFO:teuthology.orchestra.run.smithi018.stdout:Name : grub2-tools 2024-03-25T18:23:12.972 INFO:teuthology.orchestra.run.smithi018.stdout:Epoch : 1 2024-03-25T18:23:12.972 INFO:teuthology.orchestra.run.smithi018.stdout:Version : 2.06 2024-03-25T18:23:12.972 INFO:teuthology.orchestra.run.smithi018.stdout:Release : 68.el9 2024-03-25T18:23:12.972 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: x86_64 2024-03-25T18:23:12.972 INFO:teuthology.orchestra.run.smithi018.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:12.972 INFO:teuthology.orchestra.run.smithi018.stdout:Group : Unspecified 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Size : 8264254 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:License : GPLv3+ 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Packager : builder@centos.org 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Vendor : CentOS 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout:Description : 2024-03-25T18:23:12.973 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:23:12.974 INFO:teuthology.orchestra.run.smithi018.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:12.974 INFO:teuthology.orchestra.run.smithi018.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:12.974 INFO:teuthology.orchestra.run.smithi018.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:12.974 INFO:teuthology.orchestra.run.smithi018.stdout:hardware devices. 2024-03-25T18:23:12.974 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:23:12.974 INFO:teuthology.orchestra.run.smithi018.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:12.975 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:12.975 INFO:teuthology.task.kernel:Updating grub on smithi018 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:12.975 DEBUG:teuthology.orchestra.run.smithi018:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:13.518 INFO:teuthology.orchestra.run.smithi018.stderr:Generating grub configuration file ... 2024-03-25T18:23:13.933 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:13.933 INFO:teuthology.orchestra.run.smithi053.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:13.933 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:23:13.979 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:14.372 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:14.372 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:14.372 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:14.372 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:14.797 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:14.798 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:23:14.798 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:23:14.798 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:14.798 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:14.798 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:14.798 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:14.798 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:23:14.798 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:23:14.923 INFO:teuthology.orchestra.run.smithi018.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:14.954 INFO:teuthology.orchestra.run.smithi018.stderr:done 2024-03-25T18:23:14.956 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-25T18:23:14.972 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.BTuPQSuUlt 2024-03-25T18:23:14.972 DEBUG:teuthology.orchestra.run.smithi018:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BTuPQSuUlt 2024-03-25T18:23:15.043 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.BTuPQSuUlt 2024-03-25T18:23:15.108 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:15.180 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.BTuPQSuUlt is 6KB 2024-03-25T18:23:15.201 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.BTuPQSuUlt 2024-03-25T18:23:15.209 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:15.210 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:15.210 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:15.210 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:15.211 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:15.211 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:23:15.211 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:15.211 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:15.215 DEBUG:teuthology.orchestra.run.smithi018:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:15.290 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:15.290 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:15.290 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:15.290 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:15.292 DEBUG:teuthology.orchestra.run.smithi018:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:15.292 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:15.293 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:15.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.06 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 68.el9 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Group : Unspecified 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 8264254 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:15.331 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : builder@centos.org 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:23:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:15.333 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:15.333 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:15.333 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:15.441 DEBUG:teuthology.orchestra.run.smithi018:> sudo shutdown -r now 2024-03-25T18:23:15.961 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-03-25T18:23:17.517 INFO:teuthology.orchestra.run.smithi053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:17.560 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-03-25T18:23:17.570 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-03-25T18:23:17.594 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.xAK7t9QPjM 2024-03-25T18:23:17.594 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xAK7t9QPjM 2024-03-25T18:23:17.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.xAK7t9QPjM 2024-03-25T18:23:17.806 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.xAK7t9QPjM is 6KB 2024-03-25T18:23:17.855 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.xAK7t9QPjM 2024-03-25T18:23:17.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:17.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:18.100 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-03-25T18:23:45.468 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:23:45.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T18:23:45.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:23:48.106 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:23:48.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-25T18:23:48.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:03.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.18 2024-03-25T18:24:06.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-25T18:24:12.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T18:24:12.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:15.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-25T18:24:15.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:33.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-25T18:24:45.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-25T18:24:45.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:46.248 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-25T18:24:46.720 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-25T18:24:46.720 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:24:46.720 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-03-25T18:24:46.733 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:24:46.733 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:24:46.734 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:24:46.734 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:24:47.735 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:24:47.735 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:24:47.813 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-03-25T18:24:47.853 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:12.881 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:25:24.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T18:25:24.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:25.177 DEBUG:teuthology.orchestra.run.smithi018:> true 2024-03-25T18:25:25.561 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T18:25:25.561 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:25:25.561 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2024-03-25T18:25:25.576 INFO:teuthology.orchestra.run.smithi018.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:25:25.576 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:25:25.576 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:25:25.577 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:25:26.578 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:25:26.578 DEBUG:teuthology.orchestra.run.smithi018:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:25:26.642 INFO:teuthology.orchestra.run.smithi018.stdout:ttyS1 2024-03-25T18:25:26.678 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:26.678 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:25:26.686 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:25:26.686 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:25:26.689 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:25:26.709 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:25:26.716 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:25:26.726 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:25:26.726 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:25:26.746 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:25:26.779 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:25:26.786 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:25:26.787 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:25:26.810 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:25:26.847 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:26.855 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:26.861 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:26.869 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:25:26.871 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:25:26.878 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:25:26.878 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:25:26.905 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:25:26.940 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:25:26.948 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:25:26.949 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:25:26.974 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:25:26.997 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:27.071 DEBUG:teuthology.orchestra.run.smithi018:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:27.117 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:27.197 DEBUG:teuthology.orchestra.run.smithi018:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:27.242 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T18:25:27.242 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:25:27.310 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:27.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:25:27.407 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:27.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:25:27.534 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T18:25:27.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:25:27.603 DEBUG:teuthology.orchestra.run.smithi018:> sudo service rsyslog restart 2024-03-25T18:25:27.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-03-25T18:25:27.643 INFO:teuthology.orchestra.run.smithi018.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:25:27.703 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:25:28.150 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:25:28.165 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:25:28.166 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:25:28.178 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:25:28.185 DEBUG:teuthology.orchestra.run.smithi018:> sudo service auditd rotate 2024-03-25T18:25:28.255 INFO:teuthology.orchestra.run.smithi018.stdout:Rotating logs: 2024-03-25T18:25:28.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-03-25T18:25:28.315 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-03-25T18:25:28.318 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:25:28.318 DEBUG:teuthology.orchestra.run.smithi018:> /usr/sbin/getenforce 2024-03-25T18:25:28.341 INFO:teuthology.orchestra.run.smithi018.stdout:Permissive 2024-03-25T18:25:28.341 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-03-25T18:25:28.364 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-03-25T18:25:28.364 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi018.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi053.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:25:28.365 DEBUG:teuthology.orchestra.run.smithi018:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:25:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:25:28.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:25:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:25:28.437 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:25:28.437 DEBUG:teuthology.orchestra.run.smithi018:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:25:28.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:25:28.524 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:25:28.534 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:25:28.551 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:25:28.552 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi018.front.sepia.ceph.com,smithi053.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:29:53.045 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi018.front.sepia.ceph.com'), Remote(name='ubuntu@smithi053.front.sepia.ceph.com')] 2024-03-25T18:29:53.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T18:29:53.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:29:53.142 DEBUG:teuthology.orchestra.run.smithi018:> true 2024-03-25T18:29:53.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T18:29:53.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-25T18:29:53.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:29:53.319 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-25T18:29:53.398 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-25T18:29:53.399 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:29:53.410 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:29:53.411 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:29:53.411 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:29:53.413 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:29:53.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:29:53.446 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:29:53.464 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:29:53.470 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:29:53.488 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:29:53.495 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: ntpd: command not found 2024-03-25T18:29:53.510 INFO:teuthology.orchestra.run.smithi018.stdout:506 Cannot talk to daemon 2024-03-25T18:29:53.518 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-03-25T18:29:53.527 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:29:53.532 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-03-25T18:29:53.544 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:29:53.548 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:29:53.565 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:29:53.599 INFO:teuthology.orchestra.run.smithi018.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:29:53.602 INFO:teuthology.orchestra.run.smithi018.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:29:53.603 INFO:teuthology.orchestra.run.smithi018.stdout:=============================================================================== 2024-03-25T18:29:53.603 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:53.603 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:53.603 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:53.603 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:53.616 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:29:53.620 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:29:53.620 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-03-25T18:29:53.620 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:53.620 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:53.620 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:53.620 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:53.621 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:29:53.628 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:29:53.629 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:29:53.629 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:29:53.629 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:29:53.634 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:29:53.634 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:29:53.635 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:29:53.635 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:29:53.637 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:29:53.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:29:53.909 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:29:53.909 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:29:53.917 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:29:53.917 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:29:54.315 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:29:54.315 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T18:29:54.315 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:29:54.321 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:29:54.321 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T18:29:54.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:29:54.347 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:29:54.348 DEBUG:teuthology.orchestra.run.smithi018:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:29:54.352 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:29:54.352 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:29:54.420 DEBUG:teuthology.orchestra.run.smithi018:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:29:54.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:29:54.506 DEBUG:teuthology.orchestra.run.smithi018:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:29:54.507 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:29:54.535 INFO:teuthology.orchestra.run.smithi018.stdout:check_obsoletes = 1 2024-03-25T18:29:54.536 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum clean all 2024-03-25T18:29:54.574 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-03-25T18:29:54.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-03-25T18:29:54.783 INFO:teuthology.orchestra.run.smithi018.stdout:74 files removed 2024-03-25T18:29:54.811 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-radosgw 2024-03-25T18:29:54.837 INFO:teuthology.orchestra.run.smithi053.stdout:74 files removed 2024-03-25T18:29:54.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw 2024-03-25T18:29:56.068 INFO:teuthology.orchestra.run.smithi018.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-25T18:29:56.090 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 89 kB/s | 82 kB 00:00 2024-03-25T18:29:56.852 INFO:teuthology.orchestra.run.smithi018.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T18:29:56.966 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:29:57.685 INFO:teuthology.orchestra.run.smithi018.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T18:29:57.850 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T18:29:58.496 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - AppStream 8.7 MB/s | 6.7 MB 00:00 2024-03-25T18:29:58.611 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-03-25T18:30:00.071 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-25T18:30:00.202 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-25T18:30:04.425 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-03-25T18:30:04.716 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-25T18:30:09.761 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 19 MB 00:04 2024-03-25T18:30:10.043 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-03-25T18:30:16.020 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-25T18:30:17.387 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-25T18:30:17.517 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-25T18:30:18.901 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-25T18:30:20.806 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T18:30:20.952 INFO:teuthology.orchestra.run.smithi018.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 21 MB 00:03 2024-03-25T18:30:25.936 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-03-25T18:30:26.127 INFO:teuthology.orchestra.run.smithi018.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-25T18:30:28.493 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:30:28.504 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:30:28.505 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:30:28.506 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout:Install 30 Packages 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 2 Packages 2024-03-25T18:30:28.507 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:30:28.508 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 58 M 2024-03-25T18:30:28.508 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:30:28.684 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:30:28.704 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:30:28.704 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:30:28.704 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:30:28.704 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:30:28.704 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:30:28.704 INFO:teuthology.orchestra.run.smithi018.stdout:Upgrading: 2024-03-25T18:30:28.704 INFO:teuthology.orchestra.run.smithi018.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:30:28.704 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: boost-program-options 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:30:28.705 INFO:teuthology.orchestra.run.smithi018.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-argparse 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:30:28.706 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout:Installing weak dependencies: 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout:Install 30 Packages 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout:Upgrade 2 Packages 2024-03-25T18:30:28.707 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:30:28.708 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 58 M 2024-03-25T18:30:28.708 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:30:38.694 INFO:teuthology.orchestra.run.smithi018.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 569 kB/s | 5.3 MB 00:09 2024-03-25T18:30:38.810 INFO:teuthology.orchestra.run.smithi018.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T18:30:40.677 INFO:teuthology.orchestra.run.smithi018.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 397 kB/s | 742 kB 00:01 2024-03-25T18:30:42.186 INFO:teuthology.orchestra.run.smithi018.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 330 kB/s | 498 kB 00:01 2024-03-25T18:30:44.403 INFO:teuthology.orchestra.run.smithi018.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 651 kB/s | 9.7 MB 00:15 2024-03-25T18:30:44.519 INFO:teuthology.orchestra.run.smithi018.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 388 kB/s | 45 kB 00:00 2024-03-25T18:30:44.736 INFO:teuthology.orchestra.run.smithi018.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 606 kB/s | 131 kB 00:00 2024-03-25T18:30:45.061 INFO:teuthology.orchestra.run.smithi018.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 498 kB/s | 162 kB 00:00 2024-03-25T18:30:45.779 INFO:teuthology.orchestra.run.smithi018.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 447 kB/s | 320 kB 00:00 2024-03-25T18:30:46.278 INFO:teuthology.orchestra.run.smithi018.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 596 kB/s | 298 kB 00:00 2024-03-25T18:30:46.496 INFO:teuthology.orchestra.run.smithi018.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 458 kB/s | 99 kB 00:00 2024-03-25T18:30:46.604 INFO:teuthology.orchestra.run.smithi018.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 980 kB/s | 106 kB 00:00 2024-03-25T18:30:46.646 INFO:teuthology.orchestra.run.smithi018.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T18:30:46.671 INFO:teuthology.orchestra.run.smithi018.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T18:30:46.722 INFO:teuthology.orchestra.run.smithi018.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-25T18:30:46.764 INFO:teuthology.orchestra.run.smithi018.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:30:46.797 INFO:teuthology.orchestra.run.smithi018.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:30:46.861 INFO:teuthology.orchestra.run.smithi053.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 308 kB/s | 5.3 MB 00:17 2024-03-25T18:30:46.863 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:46.978 INFO:teuthology.orchestra.run.smithi053.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T18:30:46.992 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:47.058 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:47.181 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:47.216 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:47.233 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:47.286 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:47.446 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:47.798 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:47.862 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:47.884 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:48.024 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:48.113 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:48.264 INFO:teuthology.orchestra.run.smithi018.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 150 kB/s | 221 kB 00:01 2024-03-25T18:30:48.296 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:48.328 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:48.360 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:48.392 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:48.409 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:48.426 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:48.485 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:48.512 INFO:teuthology.orchestra.run.smithi053.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 484 kB/s | 742 kB 00:01 2024-03-25T18:30:48.564 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:48.650 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:48.719 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:48.736 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:48.880 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:48.971 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:49.031 INFO:teuthology.orchestra.run.smithi018.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 47 kB/s | 36 kB 00:00 2024-03-25T18:30:49.067 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:49.121 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:49.153 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:49.185 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:49.202 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:49.220 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:49.297 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:49.388 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:49.454 INFO:teuthology.orchestra.run.smithi053.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 490 kB/s | 9.7 MB 00:20 2024-03-25T18:30:49.479 INFO:teuthology.orchestra.run.smithi053.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 515 kB/s | 498 kB 00:00 2024-03-25T18:30:49.485 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:49.540 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:49.559 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:49.621 INFO:teuthology.orchestra.run.smithi053.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 318 kB/s | 45 kB 00:00 2024-03-25T18:30:49.723 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:49.816 INFO:teuthology.orchestra.run.smithi018.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:49.864 INFO:teuthology.orchestra.run.smithi018.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 57 kB/s | 47 kB 00:00 2024-03-25T18:30:50.030 INFO:teuthology.orchestra.run.smithi053.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 322 kB/s | 131 kB 00:00 2024-03-25T18:30:50.131 INFO:teuthology.orchestra.run.smithi018.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-25T18:30:50.413 INFO:teuthology.orchestra.run.smithi053.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 422 kB/s | 162 kB 00:00 2024-03-25T18:30:51.065 INFO:teuthology.orchestra.run.smithi018.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-03-25T18:30:51.132 INFO:teuthology.orchestra.run.smithi018.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-03-25T18:30:51.190 INFO:teuthology.orchestra.run.smithi018.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-03-25T18:30:51.249 INFO:teuthology.orchestra.run.smithi018.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T18:30:51.388 INFO:teuthology.orchestra.run.smithi053.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 329 kB/s | 320 kB 00:00 2024-03-25T18:30:51.408 INFO:teuthology.orchestra.run.smithi018.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-03-25T18:30:51.466 INFO:teuthology.orchestra.run.smithi018.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T18:30:51.683 INFO:teuthology.orchestra.run.smithi018.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-25T18:30:52.430 INFO:teuthology.orchestra.run.smithi053.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 286 kB/s | 298 kB 00:01 2024-03-25T18:30:53.022 INFO:teuthology.orchestra.run.smithi053.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 168 kB/s | 99 kB 00:00 2024-03-25T18:30:53.131 INFO:teuthology.orchestra.run.smithi053.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T18:30:53.173 INFO:teuthology.orchestra.run.smithi053.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T18:30:53.206 INFO:teuthology.orchestra.run.smithi053.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:30:53.273 INFO:teuthology.orchestra.run.smithi053.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T18:30:53.315 INFO:teuthology.orchestra.run.smithi053.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:30:53.349 INFO:teuthology.orchestra.run.smithi053.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:30:53.405 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:53.423 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:53.458 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:53.620 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:53.972 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:53.989 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:54.045 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:54.111 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:54.281 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:54.366 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:54.540 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:54.702 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:54.777 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:54.908 INFO:teuthology.orchestra.run.smithi053.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 142 kB/s | 221 kB 00:01 2024-03-25T18:30:54.962 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:54.981 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:54.998 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:55.079 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:55.162 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:55.179 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:55.235 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:55.267 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:55.299 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:55.333 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:55.366 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:55.528 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:55.634 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:55.683 INFO:teuthology.orchestra.run.smithi053.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 46 kB/s | 36 kB 00:00 2024-03-25T18:30:55.739 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:55.756 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:55.773 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:30:55.853 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:55.933 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:30:55.951 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:56.008 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:30:56.039 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:56.074 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:30:56.107 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:56.139 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:30:56.301 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:56.378 INFO:teuthology.orchestra.run.smithi053.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:30:56.416 INFO:teuthology.orchestra.run.smithi053.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-03-25T18:30:57.075 INFO:teuthology.orchestra.run.smithi053.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 469 kB/s | 309 kB 00:00 2024-03-25T18:30:59.033 INFO:teuthology.orchestra.run.smithi018.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 310 kB/s | 5.1 MB 00:16 2024-03-25T18:30:59.284 INFO:teuthology.orchestra.run.smithi053.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2024-03-25T18:30:59.384 INFO:teuthology.orchestra.run.smithi053.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-25T18:30:59.493 INFO:teuthology.orchestra.run.smithi053.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2024-03-25T18:30:59.593 INFO:teuthology.orchestra.run.smithi053.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-03-25T18:30:59.960 INFO:teuthology.orchestra.run.smithi053.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-25T18:31:00.102 INFO:teuthology.orchestra.run.smithi053.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-25T18:31:00.852 INFO:teuthology.orchestra.run.smithi053.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-25T18:31:01.236 INFO:teuthology.orchestra.run.smithi053.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 443 kB/s | 5.1 MB 00:11 2024-03-25T18:31:03.966 INFO:teuthology.orchestra.run.smithi018.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 584 kB/s | 20 MB 00:34 2024-03-25T18:31:04.491 INFO:teuthology.orchestra.run.smithi018.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 270 kB/s | 3.4 MB 00:12 2024-03-25T18:31:06.633 INFO:teuthology.orchestra.run.smithi018.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 422 kB/s | 3.1 MB 00:07 2024-03-25T18:31:06.640 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:06.640 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.5 MB/s | 58 MB 00:37 2024-03-25T18:31:06.672 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:31:06.714 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:31:06.715 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:31:07.208 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:31:07.208 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:31:07.669 INFO:teuthology.orchestra.run.smithi053.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 498 kB/s | 3.1 MB 00:06 2024-03-25T18:31:08.119 INFO:teuthology.orchestra.run.smithi053.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 475 kB/s | 3.4 MB 00:07 2024-03-25T18:31:08.419 INFO:teuthology.orchestra.run.smithi053.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 518 kB/s | 20 MB 00:39 2024-03-25T18:31:08.426 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:08.426 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.4 MB/s | 58 MB 00:39 2024-03-25T18:31:08.456 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:31:08.496 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:31:08.497 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:31:08.703 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:31:08.907 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:31:08.937 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:31:08.968 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:31:08.994 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:31:08.994 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:31:09.002 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:09.019 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:09.211 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:09.266 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:09.618 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:31:09.672 INFO:teuthology.orchestra.run.smithi018.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:09.764 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:09.862 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:31:09.890 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:09.953 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:09.982 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:31:10.026 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:31:10.095 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:10.117 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:10.165 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:10.200 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:31:10.309 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:31:10.350 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:31:10.391 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:31:10.423 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:31:10.452 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:31:10.493 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:31:10.513 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:31:10.599 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:31:10.640 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:31:10.666 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:31:10.688 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:10.705 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:10.824 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:31:10.865 INFO:teuthology.orchestra.run.smithi018.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:10.880 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:10.945 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:10.947 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:10.978 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:31:11.020 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:31:11.288 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:31:11.343 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:11.418 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:11.502 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:31:11.534 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:11.606 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:11.649 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:31:11.675 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:31:11.694 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:11.720 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:31:11.757 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:11.785 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:31:11.830 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:31:11.899 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:11.977 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:12.011 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:31:12.123 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:31:12.163 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:31:12.203 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:31:12.236 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:31:12.265 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:31:12.298 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:31:12.625 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:31:12.653 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:12.743 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:12.774 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:31:12.827 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:31:13.308 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:31:13.339 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:13.393 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:13.464 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:31:13.500 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:31:14.015 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:14.058 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:14.658 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:14.703 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:14.793 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:14.793 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:31:14.793 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:31:14.793 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:31:14.834 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:15.713 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:16.302 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:31:16.341 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:16.416 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:31:16.416 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:31:16.416 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:31:16.416 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:31:16.451 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:24.749 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:24.749 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /sys 2024-03-25T18:31:24.749 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /proc 2024-03-25T18:31:24.749 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /mnt 2024-03-25T18:31:24.749 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /var/tmp 2024-03-25T18:31:24.749 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /home 2024-03-25T18:31:24.750 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /root 2024-03-25T18:31:24.750 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /tmp 2024-03-25T18:31:24.750 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:31:25.662 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:25.686 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:25.686 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:31:25.687 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:31:25.687 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:25.687 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:25.687 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:31:25.688 INFO:teuthology.orchestra.run.smithi018.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:25.773 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:25.773 INFO:teuthology.orchestra.run.smithi018.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:26.273 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:26.273 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-03-25T18:31:26.274 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-03-25T18:31:26.274 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-03-25T18:31:26.274 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-03-25T18:31:26.274 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-03-25T18:31:26.274 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-03-25T18:31:26.274 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-03-25T18:31:26.274 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:31:27.013 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:31:27.014 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:31:27.015 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:31:27.016 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:31:27.197 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:27.220 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:31:27.220 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:31:27.221 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:31:27.221 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:27.221 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:31:27.221 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:31:27.222 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:27.308 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:31:27.308 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:28.098 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:28.098 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:31:28.098 INFO:teuthology.orchestra.run.smithi018.stdout:Upgraded: 2024-03-25T18:31:28.098 INFO:teuthology.orchestra.run.smithi018.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.098 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.098 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:31:28.099 INFO:teuthology.orchestra.run.smithi018.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:31:28.100 INFO:teuthology.orchestra.run.smithi018.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:31:28.101 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:31:28.101 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:31:28.441 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:28.441 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:31:28.441 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:31:28.441 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:31:28.441 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:31:28.442 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:31:28.443 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:31:28.445 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-test 2024-03-25T18:31:29.018 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:03 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:31:29.174 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:31:29.174 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:31:29.175 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:31:29.176 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:31:29.176 INFO:teuthology.orchestra.run.smithi018.stdout:Install 6 Packages 2024-03-25T18:31:29.176 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:31:29.176 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 48 M 2024-03-25T18:31:29.177 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 204 M 2024-03-25T18:31:29.177 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.403 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.404 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:31:29.405 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:31:29.518 INFO:teuthology.orchestra.run.smithi018.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 713 kB/s | 217 kB 00:00 2024-03-25T18:31:29.626 INFO:teuthology.orchestra.run.smithi018.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2024-03-25T18:31:29.669 INFO:teuthology.orchestra.run.smithi018.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T18:31:29.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-test 2024-03-25T18:31:29.752 INFO:teuthology.orchestra.run.smithi018.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-25T18:31:29.986 INFO:teuthology.orchestra.run.smithi018.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 211 kB/s | 163 kB 00:00 2024-03-25T18:31:30.287 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:05 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:31:30.456 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:31:30.457 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:31:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:31:30.457 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:31:30.457 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:31:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:31:30.457 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-03-25T18:31:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:31:30.459 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 48 M 2024-03-25T18:31:30.459 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 204 M 2024-03-25T18:31:30.459 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:31:30.772 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:31:30.822 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T18:31:30.856 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T18:31:30.998 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T18:31:31.557 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 170 kB/s | 163 kB 00:00 2024-03-25T18:33:06.240 INFO:teuthology.orchestra.run.smithi018.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 499 kB/s | 47 MB 01:37 2024-03-25T18:33:06.241 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:06.242 INFO:teuthology.orchestra.run.smithi018.stdout:Total 508 kB/s | 48 MB 01:37 2024-03-25T18:33:06.253 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:33:06.273 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:33:06.273 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:33:06.670 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:33:06.670 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:33:07.267 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:33:07.349 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:33:07.396 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:33:07.435 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:07.492 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:33:07.517 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:12.288 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:12.378 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:14.687 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:14.687 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:33:14.687 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:33:14.688 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:14.688 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:33:14.688 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:33:15.095 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:33:15.096 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:33:15.415 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph 2024-03-25T18:33:15.987 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:02:49 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:33:16.155 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:33:16.158 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-25T18:33:16.158 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:33:16.158 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-25T18:33:16.158 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:33:16.158 INFO:teuthology.orchestra.run.smithi018.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:33:16.158 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:33:16.158 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:33:16.159 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:33:16.160 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout:Install 36 Packages 2024-03-25T18:33:16.161 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:33:16.162 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 30 M 2024-03-25T18:33:16.162 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 107 M 2024-03-25T18:33:16.162 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:33:16.612 INFO:teuthology.orchestra.run.smithi018.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 20 kB/s | 6.6 kB 00:00 2024-03-25T18:33:21.037 INFO:teuthology.orchestra.run.smithi018.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 321 kB/s | 1.5 MB 00:04 2024-03-25T18:33:21.662 INFO:teuthology.orchestra.run.smithi018.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 404 kB/s | 2.1 MB 00:05 2024-03-25T18:33:22.004 INFO:teuthology.orchestra.run.smithi018.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 724 kB/s | 247 kB 00:00 2024-03-25T18:33:22.020 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 434 kB/s | 47 MB 01:51 2024-03-25T18:33:22.022 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:22.022 INFO:teuthology.orchestra.run.smithi053.stdout:Total 442 kB/s | 48 MB 01:51 2024-03-25T18:33:22.033 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:33:22.054 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:33:22.055 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:33:22.196 INFO:teuthology.orchestra.run.smithi018.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 975 kB/s | 187 kB 00:00 2024-03-25T18:33:22.246 INFO:teuthology.orchestra.run.smithi018.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T18:33:22.288 INFO:teuthology.orchestra.run.smithi018.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-03-25T18:33:22.330 INFO:teuthology.orchestra.run.smithi018.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T18:33:22.372 INFO:teuthology.orchestra.run.smithi018.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 932 kB/s | 39 kB 00:00 2024-03-25T18:33:22.430 INFO:teuthology.orchestra.run.smithi018.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T18:33:22.447 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:33:22.448 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:33:22.497 INFO:teuthology.orchestra.run.smithi018.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T18:33:22.539 INFO:teuthology.orchestra.run.smithi018.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T18:33:22.581 INFO:teuthology.orchestra.run.smithi018.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:33:22.623 INFO:teuthology.orchestra.run.smithi018.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:33:23.062 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:33:23.081 INFO:teuthology.orchestra.run.smithi018.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 272 kB/s | 125 kB 00:00 2024-03-25T18:33:23.137 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:33:23.174 INFO:teuthology.orchestra.run.smithi018.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-25T18:33:23.174 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:33:23.215 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:23.270 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:33:23.306 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:23.499 INFO:teuthology.orchestra.run.smithi018.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 465 kB/s | 151 kB 00:00 2024-03-25T18:33:23.566 INFO:teuthology.orchestra.run.smithi018.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 651 kB/s | 43 kB 00:00 2024-03-25T18:33:23.666 INFO:teuthology.orchestra.run.smithi018.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T18:33:23.783 INFO:teuthology.orchestra.run.smithi018.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-25T18:33:23.833 INFO:teuthology.orchestra.run.smithi018.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-25T18:33:23.892 INFO:teuthology.orchestra.run.smithi018.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 304 kB/s | 18 kB 00:00 2024-03-25T18:33:23.951 INFO:teuthology.orchestra.run.smithi018.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 398 kB/s | 23 kB 00:00 2024-03-25T18:33:24.018 INFO:teuthology.orchestra.run.smithi018.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-25T18:33:24.110 INFO:teuthology.orchestra.run.smithi018.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-25T18:33:24.177 INFO:teuthology.orchestra.run.smithi018.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 695 kB/s | 46 kB 00:00 2024-03-25T18:33:24.244 INFO:teuthology.orchestra.run.smithi018.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-25T18:33:24.311 INFO:teuthology.orchestra.run.smithi018.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T18:33:24.378 INFO:teuthology.orchestra.run.smithi018.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-25T18:33:24.444 INFO:teuthology.orchestra.run.smithi018.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-25T18:33:24.512 INFO:teuthology.orchestra.run.smithi018.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 536 kB/s | 36 kB 00:00 2024-03-25T18:33:24.587 INFO:teuthology.orchestra.run.smithi018.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-25T18:33:24.662 INFO:teuthology.orchestra.run.smithi018.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T18:33:24.729 INFO:teuthology.orchestra.run.smithi018.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-25T18:33:28.039 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:33:28.054 INFO:teuthology.orchestra.run.smithi018.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 419 kB/s | 4.7 MB 00:11 2024-03-25T18:33:28.124 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:33:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:33:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:33:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:33:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:33:30.967 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:33:30.967 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:33:30.968 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:33:31.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph 2024-03-25T18:33:31.846 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:03:06 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:33:32.011 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:33:32.013 INFO:teuthology.orchestra.run.smithi053.stdout:========================================================================================= 2024-03-25T18:33:32.013 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:33:32.013 INFO:teuthology.orchestra.run.smithi053.stdout:========================================================================================= 2024-03-25T18:33:32.013 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:33:32.013 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:33:32.014 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:33:32.015 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout:========================================================================================= 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout:Install 36 Packages 2024-03-25T18:33:32.016 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:33:32.017 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 30 M 2024-03-25T18:33:32.017 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 107 M 2024-03-25T18:33:32.017 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:33:32.672 INFO:teuthology.orchestra.run.smithi053.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 16 kB/s | 6.6 kB 00:00 2024-03-25T18:33:35.914 INFO:teuthology.orchestra.run.smithi053.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 418 kB/s | 1.5 MB 00:03 2024-03-25T18:33:37.415 INFO:teuthology.orchestra.run.smithi053.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 4.7 MB 00:04 2024-03-25T18:33:37.614 INFO:teuthology.orchestra.run.smithi053.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 407 kB/s | 2.1 MB 00:05 2024-03-25T18:33:37.740 INFO:teuthology.orchestra.run.smithi053.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:33:37.765 INFO:teuthology.orchestra.run.smithi053.stdout:(6/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 706 kB/s | 247 kB 00:00 2024-03-25T18:33:37.790 INFO:teuthology.orchestra.run.smithi053.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T18:33:37.815 INFO:teuthology.orchestra.run.smithi053.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 634 kB/s | 32 kB 00:00 2024-03-25T18:33:37.840 INFO:teuthology.orchestra.run.smithi053.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-25T18:33:37.866 INFO:teuthology.orchestra.run.smithi053.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-25T18:33:37.908 INFO:teuthology.orchestra.run.smithi053.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-25T18:33:37.966 INFO:teuthology.orchestra.run.smithi053.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-25T18:33:37.991 INFO:teuthology.orchestra.run.smithi053.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-25T18:33:38.017 INFO:teuthology.orchestra.run.smithi053.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T18:33:38.051 INFO:teuthology.orchestra.run.smithi053.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-25T18:33:38.142 INFO:teuthology.orchestra.run.smithi053.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-03-25T18:33:38.484 INFO:teuthology.orchestra.run.smithi053.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 267 kB/s | 125 kB 00:00 2024-03-25T18:33:38.801 INFO:teuthology.orchestra.run.smithi053.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 229 kB/s | 151 kB 00:00 2024-03-25T18:33:39.018 INFO:teuthology.orchestra.run.smithi053.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 81 kB/s | 43 kB 00:00 2024-03-25T18:33:39.060 INFO:teuthology.orchestra.run.smithi053.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 664 kB/s | 172 kB 00:00 2024-03-25T18:33:39.160 INFO:teuthology.orchestra.run.smithi053.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T18:33:39.261 INFO:teuthology.orchestra.run.smithi053.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 177 kB/s | 18 kB 00:00 2024-03-25T18:33:39.361 INFO:teuthology.orchestra.run.smithi053.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-25T18:33:39.494 INFO:teuthology.orchestra.run.smithi053.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 146 kB/s | 19 kB 00:00 2024-03-25T18:33:39.528 INFO:teuthology.orchestra.run.smithi053.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 703 kB/s | 358 kB 00:00 2024-03-25T18:33:39.587 INFO:teuthology.orchestra.run.smithi053.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-25T18:33:39.629 INFO:teuthology.orchestra.run.smithi053.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2024-03-25T18:33:39.704 INFO:teuthology.orchestra.run.smithi053.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 674 kB/s | 79 kB 00:00 2024-03-25T18:33:39.812 INFO:teuthology.orchestra.run.smithi053.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2024-03-25T18:33:39.996 INFO:teuthology.orchestra.run.smithi053.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 490 kB/s | 90 kB 00:00 2024-03-25T18:33:40.057 INFO:teuthology.orchestra.run.smithi053.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 639 kB/s | 272 kB 00:00 2024-03-25T18:33:40.097 INFO:teuthology.orchestra.run.smithi053.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-25T18:33:40.422 INFO:teuthology.orchestra.run.smithi053.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 625 kB/s | 230 kB 00:00 2024-03-25T18:33:40.531 INFO:teuthology.orchestra.run.smithi053.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-25T18:33:40.698 INFO:teuthology.orchestra.run.smithi053.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 711 kB/s | 427 kB 00:00 2024-03-25T18:33:53.526 INFO:teuthology.orchestra.run.smithi018.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 529 kB/s | 17 MB 00:32 2024-03-25T18:33:53.531 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:53.531 INFO:teuthology.orchestra.run.smithi018.stdout:Total 813 kB/s | 30 MB 00:37 2024-03-25T18:33:53.555 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:33:53.600 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:33:53.600 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:33:54.026 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:33:54.026 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:33:55.380 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:33:55.453 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:33:55.487 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:33:55.514 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:33:55.589 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:33:55.630 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:33:55.671 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:33:55.776 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:33:55.862 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:33:55.909 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:33:55.983 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:33:56.043 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:33:56.080 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:33:56.131 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:33:56.194 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:33:56.392 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:33:56.443 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:33:56.482 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:33:56.516 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:33:56.546 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:33:56.627 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:33:56.665 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:33:56.698 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:33:56.739 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:33:56.782 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:33:56.816 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:33:56.912 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:33:56.950 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:33:57.026 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:33:57.184 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:33:57.314 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:33:57.484 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:33:57.513 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:33:57.538 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:33:57.538 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:57.538 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:33:57.538 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:33:57.538 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:33:57.539 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:33:59.072 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:33:59.098 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:33:59.098 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:59.098 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:33:59.098 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:33:59.099 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:33:59.099 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:33:59.564 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:33:59.587 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:33:59.587 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:59.587 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:33:59.587 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:33:59.587 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:33:59.587 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:33:59.807 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:33:59.829 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:33:59.830 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:59.830 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:33:59.830 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:33:59.830 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:33:59.830 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:33:59.898 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:01.492 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:01.492 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:34:01.492 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:34:01.492 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:34:01.492 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:34:01.492 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:34:01.493 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:01.494 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:34:01.495 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:34:01.495 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:34:01.495 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:34:01.495 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:34:02.711 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:34:02.711 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:02.711 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:34:02.711 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:34:02.712 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:34:02.713 INFO:teuthology.orchestra.run.smithi018.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:34:02.714 INFO:teuthology.orchestra.run.smithi018.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:34:02.714 INFO:teuthology.orchestra.run.smithi018.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:34:02.714 INFO:teuthology.orchestra.run.smithi018.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:34:02.714 INFO:teuthology.orchestra.run.smithi018.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:34:02.714 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:02.714 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:03.131 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-base 2024-03-25T18:34:03.737 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:03:37 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:03.838 INFO:teuthology.orchestra.run.smithi018.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:03.893 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:03.895 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:34:03.895 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:03.975 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install cephadm 2024-03-25T18:34:04.542 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:03:38 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:04.698 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:04.699 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:04.699 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:34:04.699 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 747 k 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 754 k 2024-03-25T18:34:04.700 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:34:04.854 INFO:teuthology.orchestra.run.smithi053.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 594 kB/s | 17 MB 00:28 2024-03-25T18:34:04.860 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:04.860 INFO:teuthology.orchestra.run.smithi053.stdout:Total 925 kB/s | 30 MB 00:32 2024-03-25T18:34:04.886 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:34:04.932 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:34:04.932 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:34:05.353 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:34:05.354 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:34:06.718 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:34:06.806 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:34:06.843 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:34:06.884 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:34:06.957 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:34:07.008 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:34:07.040 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:34:07.135 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:34:07.138 INFO:teuthology.orchestra.run.smithi018.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 307 kB/s | 747 kB 00:02 2024-03-25T18:34:07.139 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:07.139 INFO:teuthology.orchestra.run.smithi018.stdout:Total 306 kB/s | 747 kB 00:02 2024-03-25T18:34:07.139 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:34:07.146 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:34:07.146 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:34:07.153 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:34:07.153 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:34:07.214 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:34:07.233 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:34:07.260 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:34:07.335 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:34:07.387 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:34:07.433 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:34:07.491 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:34:07.554 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:34:07.693 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:07.747 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:34:07.770 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:07.794 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:34:07.841 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:34:07.876 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:34:07.906 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:34:07.984 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:34:08.016 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:08.065 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:34:08.115 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:34:08.122 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:08.150 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:34:08.184 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:34:08.279 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:34:08.315 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:34:08.400 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:34:08.419 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:08.419 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:08.419 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:34:08.419 INFO:teuthology.orchestra.run.smithi018.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:08.419 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:08.419 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:08.546 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:34:08.634 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:34:08.672 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:08.839 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:34:08.876 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:08.902 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:08.902 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:08.902 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:34:08.902 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:08.902 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:08.902 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:09.208 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:03:43 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:09.366 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:09.366 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:09.366 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T18:34:09.366 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:09.366 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:34:09.366 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:34:09.367 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:09.367 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:34:09.367 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:09.367 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:34:09.367 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:09.367 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 145 k 2024-03-25T18:34:09.368 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 436 k 2024-03-25T18:34:09.368 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:34:10.053 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 212 kB/s | 145 kB 00:00 2024-03-25T18:34:10.054 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:10.054 INFO:teuthology.orchestra.run.smithi018.stdout:Total 211 kB/s | 145 kB 00:00 2024-03-25T18:34:10.055 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:34:10.063 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:34:10.064 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:34:10.119 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:34:10.119 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:34:10.398 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:34:10.435 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:10.461 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:10.462 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:10.462 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:34:10.462 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:10.462 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:10.462 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:10.487 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:10.511 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:10.511 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:10.511 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:34:10.511 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:10.936 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:10.959 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:10.959 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:10.959 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:34:10.959 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:10.959 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:10.959 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:11.178 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:11.201 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:11.201 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:11.201 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:34:11.201 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:11.201 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:11.201 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:11.270 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:11.486 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:11.486 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:11.486 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:34:11.486 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:11.486 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:11.486 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:11.775 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr 2024-03-25T18:34:12.348 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:03:46 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:12.447 INFO:teuthology.orchestra.run.smithi018.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:12.503 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:12.504 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:34:12.504 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:12.587 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:34:12.852 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:12.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:34:12.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:34:12.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:34:12.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:34:12.852 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:34:12.853 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:34:12.854 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:34:12.855 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:34:13.157 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:03:47 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:13.312 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:13.313 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-25T18:34:13.313 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:34:13.313 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:34:13.314 INFO:teuthology.orchestra.run.smithi018.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:34:13.315 INFO:teuthology.orchestra.run.smithi018.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:34:13.315 INFO:teuthology.orchestra.run.smithi018.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:34:13.315 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:13.315 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:34:13.315 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-25T18:34:13.315 INFO:teuthology.orchestra.run.smithi018.stdout:Install 12 Packages 2024-03-25T18:34:13.315 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:13.315 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 6.9 M 2024-03-25T18:34:13.316 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 26 M 2024-03-25T18:34:13.316 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:13.798 INFO:teuthology.orchestra.run.smithi053.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:34:13.799 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:13.800 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:34:14.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base 2024-03-25T18:34:14.233 INFO:teuthology.orchestra.run.smithi018.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 70 kB/s | 31 kB 00:00 2024-03-25T18:34:14.333 INFO:teuthology.orchestra.run.smithi018.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 28 kB/s | 15 kB 00:00 2024-03-25T18:34:14.392 INFO:teuthology.orchestra.run.smithi018.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-25T18:34:14.417 INFO:teuthology.orchestra.run.smithi018.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T18:34:14.742 INFO:teuthology.orchestra.run.smithi018.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-03-25T18:34:14.751 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:03:49 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:34:14.852 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:14.906 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:34:14.907 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:34:14.907 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:34:15.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install cephadm 2024-03-25T18:34:15.134 INFO:teuthology.orchestra.run.smithi018.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 50 kB/s | 19 kB 00:00 2024-03-25T18:34:15.159 INFO:teuthology.orchestra.run.smithi018.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 738 kB/s | 548 kB 00:00 2024-03-25T18:34:15.293 INFO:teuthology.orchestra.run.smithi018.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T18:34:15.426 INFO:teuthology.orchestra.run.smithi018.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 231 kB/s | 31 kB 00:00 2024-03-25T18:34:15.551 INFO:teuthology.orchestra.run.smithi018.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-25T18:34:15.664 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:03:50 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:34:15.810 INFO:teuthology.orchestra.run.smithi018.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.0 MB/s | 2.0 MB 00:00 2024-03-25T18:34:15.819 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:34:15.820 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:34:15.820 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:34:15.820 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:34:15.820 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 747 k 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 754 k 2024-03-25T18:34:15.821 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:34:18.143 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 322 kB/s | 747 kB 00:02 2024-03-25T18:34:18.143 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:18.144 INFO:teuthology.orchestra.run.smithi053.stdout:Total 322 kB/s | 747 kB 00:02 2024-03-25T18:34:18.144 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:34:18.150 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:34:18.150 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:34:18.157 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:34:18.157 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:34:18.232 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:34:18.318 INFO:teuthology.orchestra.run.smithi018.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 402 kB/s | 1.8 MB 00:04 2024-03-25T18:34:18.324 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:18.324 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.4 MB/s | 6.9 MB 00:05 2024-03-25T18:34:18.344 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:34:18.363 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:34:18.363 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:34:18.559 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:34:18.560 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:34:18.621 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:18.709 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:19.074 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:34:19.144 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:34:19.179 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:34:19.226 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:34:19.256 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:34:19.268 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:19.369 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:34:19.503 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:34:19.556 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:19.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:19.556 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:34:19.556 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:19.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:19.557 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:34:19.595 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:34:19.644 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:34:19.672 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:34:19.704 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:34:19.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:34:19.957 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:34:19.987 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:34:20.345 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:03:55 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:34:20.500 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:34:20.500 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:34:20.500 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T18:34:20.501 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:34:20.501 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:34:20.501 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:34:20.501 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:20.501 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:34:20.501 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:34:20.501 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T18:34:20.501 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:20.502 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 145 k 2024-03-25T18:34:20.502 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 436 k 2024-03-25T18:34:20.502 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:34:20.789 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:34:20.790 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:34:20.790 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:34:21.143 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:34:21.144 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:34:21.144 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:34:21.144 INFO:teuthology.orchestra.run.smithi018.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:34:21.144 INFO:teuthology.orchestra.run.smithi018.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:34:21.144 INFO:teuthology.orchestra.run.smithi018.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:34:21.144 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:21.144 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:21.423 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:34:21.490 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 147 kB/s | 145 kB 00:00 2024-03-25T18:34:21.490 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:21.491 INFO:teuthology.orchestra.run.smithi053.stdout:Total 147 kB/s | 145 kB 00:00 2024-03-25T18:34:21.491 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:34:21.500 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:34:21.500 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:34:21.558 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:34:21.559 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:34:21.708 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:34:21.782 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:21.807 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:21.807 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:21.807 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:34:21.807 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:21.987 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:03:55 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:22.143 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:34:22.145 INFO:teuthology.orchestra.run.smithi018.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout:Install 12 Packages 2024-03-25T18:34:22.146 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:22.147 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 37 M 2024-03-25T18:34:22.147 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 209 M 2024-03-25T18:34:22.147 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:34:22.788 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:34:22.788 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:22.788 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:34:22.788 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:22.788 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:22.788 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:34:22.841 INFO:teuthology.orchestra.run.smithi018.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-25T18:34:22.875 INFO:teuthology.orchestra.run.smithi018.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 516 kB/s | 17 kB 00:00 2024-03-25T18:34:22.917 INFO:teuthology.orchestra.run.smithi018.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 819 kB/s | 34 kB 00:00 2024-03-25T18:34:23.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr 2024-03-25T18:34:23.184 INFO:teuthology.orchestra.run.smithi018.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-25T18:34:23.351 INFO:teuthology.orchestra.run.smithi018.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-25T18:34:23.409 INFO:teuthology.orchestra.run.smithi018.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.2 MB/s | 3.0 MB 00:00 2024-03-25T18:34:23.434 INFO:teuthology.orchestra.run.smithi018.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T18:34:23.610 INFO:teuthology.orchestra.run.smithi018.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T18:34:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:03:58 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:34:23.660 INFO:teuthology.orchestra.run.smithi018.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-25T18:34:23.729 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:23.785 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:34:23.786 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:34:23.786 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:34:23.810 INFO:teuthology.orchestra.run.smithi018.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-25T18:34:23.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:34:23.994 INFO:teuthology.orchestra.run.smithi018.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 738 kB/s | 245 kB 00:00 2024-03-25T18:34:24.429 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:03:59 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:34:24.585 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:34:24.586 INFO:teuthology.orchestra.run.smithi053.stdout:====================================================================================== 2024-03-25T18:34:24.586 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:34:24.586 INFO:teuthology.orchestra.run.smithi053.stdout:====================================================================================== 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:34:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:34:24.588 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:34:24.588 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:34:24.588 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:24.588 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:34:24.588 INFO:teuthology.orchestra.run.smithi053.stdout:====================================================================================== 2024-03-25T18:34:24.588 INFO:teuthology.orchestra.run.smithi053.stdout:Install 12 Packages 2024-03-25T18:34:24.588 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:24.589 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 6.9 M 2024-03-25T18:34:24.589 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 26 M 2024-03-25T18:34:24.589 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:34:25.301 INFO:teuthology.orchestra.run.smithi053.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 74 kB/s | 31 kB 00:00 2024-03-25T18:34:25.327 INFO:teuthology.orchestra.run.smithi053.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 34 kB/s | 15 kB 00:00 2024-03-25T18:34:25.510 INFO:teuthology.orchestra.run.smithi053.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-25T18:34:25.569 INFO:teuthology.orchestra.run.smithi053.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-03-25T18:34:25.686 INFO:teuthology.orchestra.run.smithi053.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-03-25T18:34:25.953 INFO:teuthology.orchestra.run.smithi053.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-25T18:34:26.278 INFO:teuthology.orchestra.run.smithi053.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 773 kB/s | 548 kB 00:00 2024-03-25T18:34:26.720 INFO:teuthology.orchestra.run.smithi053.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 2.7 MB/s | 2.0 MB 00:00 2024-03-25T18:34:26.753 INFO:teuthology.orchestra.run.smithi053.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 304 kB/s | 144 kB 00:00 2024-03-25T18:34:26.979 INFO:teuthology.orchestra.run.smithi053.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 119 kB/s | 31 kB 00:00 2024-03-25T18:34:27.254 INFO:teuthology.orchestra.run.smithi053.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 377 kB/s | 188 kB 00:00 2024-03-25T18:34:28.671 INFO:teuthology.orchestra.run.smithi053.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 481 kB/s | 1.8 MB 00:03 2024-03-25T18:34:28.675 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:28.675 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.7 MB/s | 6.9 MB 00:04 2024-03-25T18:34:28.695 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:34:28.715 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:34:28.715 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:34:28.905 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:34:28.906 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:34:29.437 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:34:29.530 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:34:29.573 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:34:29.622 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:34:29.658 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:34:29.758 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:34:29.896 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:34:29.989 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:34:30.039 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:34:30.067 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:34:30.098 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:34:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:34:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:34:31.179 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:34:31.179 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:34:31.179 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:34:31.179 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:34:31.179 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:34:31.179 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:34:31.179 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:34:31.180 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:34:31.180 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:34:31.180 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:34:31.180 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:34:31.180 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:34:31.531 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:34:31.532 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:34:31.532 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:34:31.532 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:34:31.532 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:34:31.532 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:31.532 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:34:31.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:34:32.332 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:04:07 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:34:32.489 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:34:32.490 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-03-25T18:34:32.490 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:34:32.491 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout:Install 12 Packages 2024-03-25T18:34:32.492 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:34:32.493 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 37 M 2024-03-25T18:34:32.493 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 209 M 2024-03-25T18:34:32.493 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:34:32.935 INFO:teuthology.orchestra.run.smithi053.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 264 kB/s | 32 kB 00:00 2024-03-25T18:34:32.994 INFO:teuthology.orchestra.run.smithi053.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 505 kB/s | 17 kB 00:00 2024-03-25T18:34:33.027 INFO:teuthology.orchestra.run.smithi053.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:34:33.186 INFO:teuthology.orchestra.run.smithi053.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T18:34:33.341 INFO:teuthology.orchestra.run.smithi053.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 4.8 MB 00:00 2024-03-25T18:34:33.395 INFO:teuthology.orchestra.run.smithi053.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-25T18:34:33.428 INFO:teuthology.orchestra.run.smithi053.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.5 MB/s | 217 kB 00:00 2024-03-25T18:34:33.612 INFO:teuthology.orchestra.run.smithi053.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-25T18:34:33.771 INFO:teuthology.orchestra.run.smithi053.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T18:34:33.796 INFO:teuthology.orchestra.run.smithi053.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-25T18:34:33.913 INFO:teuthology.orchestra.run.smithi053.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-25T18:34:36.209 INFO:teuthology.orchestra.run.smithi018.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 560 kB/s | 7.4 MB 00:13 2024-03-25T18:34:36.211 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:36.212 INFO:teuthology.orchestra.run.smithi018.stdout:Total 2.6 MB/s | 37 MB 00:14 2024-03-25T18:34:36.254 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:34:36.281 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:34:36.281 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:34:36.659 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:34:36.659 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:34:37.379 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:34:37.449 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:34:37.480 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:34:37.586 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:37.617 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:34:37.793 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:37.816 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:37.898 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:34:38.587 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:34:38.707 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:34:40.020 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:34:41.095 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:34:41.123 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:43.780 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:43.780 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:34:43.780 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:34:43.780 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:43.780 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:34:43.781 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:43.781 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:43.781 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:34:43.781 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:34:43.781 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:34:43.781 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:34:43.781 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:34:44.207 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:34:44.208 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:34:44.208 INFO:teuthology.orchestra.run.smithi018.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:34:44.208 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:44.208 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:44.589 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:34:45.166 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:45.323 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:45.324 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:34:45.324 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:34:45.324 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:34:45.324 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:34:45.324 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:34:45.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:34:45.326 INFO:teuthology.orchestra.run.smithi018.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:34:45.326 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:45.326 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:34:45.326 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:34:45.326 INFO:teuthology.orchestra.run.smithi018.stdout:Install 13 Packages 2024-03-25T18:34:45.326 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:45.326 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 2.0 M 2024-03-25T18:34:45.327 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 26 M 2024-03-25T18:34:45.327 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:34:45.524 INFO:teuthology.orchestra.run.smithi018.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 214 kB/s | 24 kB 00:00 2024-03-25T18:34:45.574 INFO:teuthology.orchestra.run.smithi018.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-03-25T18:34:45.625 INFO:teuthology.orchestra.run.smithi018.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:34:45.641 INFO:teuthology.orchestra.run.smithi018.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:34:45.684 INFO:teuthology.orchestra.run.smithi018.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T18:34:45.700 INFO:teuthology.orchestra.run.smithi018.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 742 kB/s | 43 kB 00:00 2024-03-25T18:34:45.909 INFO:teuthology.orchestra.run.smithi018.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2024-03-25T18:34:45.993 INFO:teuthology.orchestra.run.smithi018.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-03-25T18:34:46.135 INFO:teuthology.orchestra.run.smithi018.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 899 kB/s | 203 kB 00:00 2024-03-25T18:34:46.160 INFO:teuthology.orchestra.run.smithi018.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 70 kB/s | 52 kB 00:00 2024-03-25T18:34:46.252 INFO:teuthology.orchestra.run.smithi018.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-03-25T18:34:46.477 INFO:teuthology.orchestra.run.smithi018.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 283 kB/s | 90 kB 00:00 2024-03-25T18:34:46.636 INFO:teuthology.orchestra.run.smithi018.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.6 MB/s | 1.0 MB 00:00 2024-03-25T18:34:46.638 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:46.638 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-25T18:34:46.639 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:34:46.654 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:34:46.655 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:34:46.757 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:34:46.758 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:34:47.290 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:34:47.472 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:34:47.521 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:34:47.576 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:34:47.605 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:34:47.647 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:34:47.742 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:34:47.852 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:34:47.906 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:34:48.664 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:34:48.720 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:34:48.753 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:34:48.811 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:34:48.840 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:34:49.624 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:34:49.624 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:34:49.624 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:34:49.625 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:34:49.991 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:34:49.992 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:34:49.992 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:34:49.992 INFO:teuthology.orchestra.run.smithi018.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:34:49.992 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:49.992 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:50.248 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:34:50.823 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:50.979 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:50.980 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:34:50.980 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:34:50.980 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:34:50.980 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout:Install 6 Packages 2024-03-25T18:34:50.981 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:50.982 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 6.8 M 2024-03-25T18:34:50.982 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 33 M 2024-03-25T18:34:50.982 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:34:51.639 INFO:teuthology.orchestra.run.smithi018.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-25T18:34:51.940 INFO:teuthology.orchestra.run.smithi018.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-03-25T18:34:52.332 INFO:teuthology.orchestra.run.smithi018.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 792 kB/s | 548 kB 00:00 2024-03-25T18:34:52.365 INFO:teuthology.orchestra.run.smithi018.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 136 kB/s | 58 kB 00:00 2024-03-25T18:34:52.449 INFO:teuthology.orchestra.run.smithi018.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 437 kB/s | 51 kB 00:00 2024-03-25T18:34:52.874 INFO:teuthology.orchestra.run.smithi018.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 98 kB/s | 138 kB 00:01 2024-03-25T18:34:52.878 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:52.878 INFO:teuthology.orchestra.run.smithi018.stdout:Total 3.6 MB/s | 6.8 MB 00:01 2024-03-25T18:34:52.893 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:34:52.904 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:34:52.904 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:34:53.003 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:34:53.004 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:34:53.310 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:34:53.479 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:34:53.534 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:34:53.561 INFO:teuthology.orchestra.run.smithi053.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 365 kB/s | 7.4 MB 00:20 2024-03-25T18:34:53.563 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:53.564 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.8 MB/s | 37 MB 00:21 2024-03-25T18:34:53.608 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:34:53.634 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:34:53.635 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:34:54.009 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:34:54.009 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:34:54.080 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:34:54.163 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:34:54.225 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:34:54.254 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:34:54.744 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:34:54.826 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:34:54.881 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:34:54.980 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:55.018 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:34:55.041 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:34:55.041 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:34:55.041 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:34:55.041 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:34:55.041 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:34:55.041 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:34:55.205 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:55.234 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:34:55.325 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:34:55.325 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:55.325 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:34:55.325 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:55.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:34:55.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:34:55.325 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:34:55.326 INFO:teuthology.orchestra.run.smithi018.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:34:55.326 INFO:teuthology.orchestra.run.smithi018.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:34:55.326 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:55.326 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:34:55.571 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-fuse 2024-03-25T18:34:56.000 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:34:56.117 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:34:56.144 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:30 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:34:56.300 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:34:56.300 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:56.300 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout:Install 2 Packages 2024-03-25T18:34:56.301 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:34:56.302 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 964 k 2024-03-25T18:34:56.302 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.8 M 2024-03-25T18:34:56.302 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:34:56.409 INFO:teuthology.orchestra.run.smithi018.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 736 kB/s | 79 kB 00:00 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:34:58.513 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:34:58.540 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:58.734 INFO:teuthology.orchestra.run.smithi018.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 364 kB/s | 885 kB 00:02 2024-03-25T18:34:58.735 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:58.736 INFO:teuthology.orchestra.run.smithi018.stdout:Total 396 kB/s | 964 kB 00:02 2024-03-25T18:34:58.736 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:34:58.760 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:34:58.760 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:34:58.840 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:34:58.840 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:34:59.148 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:34:59.280 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:34:59.461 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:00.129 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:00.129 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:35:00.410 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:35:00.410 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:00.410 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:35:00.410 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:35:00.410 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:00.410 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:00.712 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-volume 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:35:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:35:01.288 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:35 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:01.447 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:01.447 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:01.447 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T18:35:01.447 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:01.447 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:35:01.447 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:35:01.448 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:35:01.448 INFO:teuthology.orchestra.run.smithi018.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:35:01.448 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:01.448 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:35:01.448 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:01.448 INFO:teuthology.orchestra.run.smithi018.stdout:Install 2 Packages 2024-03-25T18:35:01.448 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:01.449 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 332 k 2024-03-25T18:35:01.449 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.5 M 2024-03-25T18:35:01.449 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:35:01.555 INFO:teuthology.orchestra.run.smithi018.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 644 kB/s | 69 kB 00:00 2024-03-25T18:35:01.668 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:35:01.668 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:01.668 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:01.668 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:01.669 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:35:01.670 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:01.670 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:02.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:35:02.606 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:04:37 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:02.639 INFO:teuthology.orchestra.run.smithi018.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 222 kB/s | 264 kB 00:01 2024-03-25T18:35:02.641 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:02.641 INFO:teuthology.orchestra.run.smithi018.stdout:Total 278 kB/s | 332 kB 00:01 2024-03-25T18:35:02.642 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:35:02.652 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:35:02.652 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:35:02.694 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:35:02.695 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:35:02.763 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:35:02.765 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:02.766 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T18:35:02.767 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2024-03-25T18:35:02.767 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:02.768 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.0 M 2024-03-25T18:35:02.768 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 26 M 2024-03-25T18:35:02.768 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:02.852 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:35:02.907 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 192 kB/s | 18 kB 00:00 2024-03-25T18:35:02.949 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 179 kB/s | 24 kB 00:00 2024-03-25T18:35:02.999 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T18:35:03.024 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:35:03.049 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:35:03.050 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:35:03.075 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 864 kB/s | 43 kB 00:00 2024-03-25T18:35:03.124 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:03.140 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:03.140 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:03.141 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:35:03.141 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:03.251 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 86 kB/s | 15 kB 00:00 2024-03-25T18:35:03.276 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2024-03-25T18:35:03.343 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 99 kB/s | 52 kB 00:00 2024-03-25T18:35:03.401 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-03-25T18:35:03.493 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-03-25T18:35:03.527 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-03-25T18:35:03.552 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-03-25T18:35:03.556 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:03.557 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-25T18:35:03.557 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:03.570 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:03.570 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:03.665 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:03.665 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:03.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:35:04.183 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:04.228 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:35:04.228 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:04.228 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:35:04.228 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:04.228 INFO:teuthology.orchestra.run.smithi018.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:35:04.228 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:04.228 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:04.372 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:35:04.429 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:35:04.484 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:35:04.521 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:35:04.562 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:35:04.581 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados-devel 2024-03-25T18:35:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:35:04.768 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:35:04.822 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:35:05.161 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:05.316 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:35:05.317 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:05.318 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 127 k 2024-03-25T18:35:05.318 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 456 k 2024-03-25T18:35:05.318 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:35:05.590 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:35:05.644 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:35:05.677 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:35:05.726 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:35:05.764 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:35:06.561 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:35:06.562 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:35:06.562 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:35:06.562 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:35:06.562 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:35:06.586 INFO:teuthology.orchestra.run.smithi018.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 100 kB/s | 127 kB 00:01 2024-03-25T18:35:06.587 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:06.587 INFO:teuthology.orchestra.run.smithi018.stdout:Total 100 kB/s | 127 kB 00:01 2024-03-25T18:35:06.587 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:35:06.594 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:35:06.594 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:35:06.657 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:35:06.657 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:35:06.806 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:35:06.912 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:35:06.929 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:35:06.930 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:35:06.930 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:35:06.930 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:35:06.930 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:35:06.930 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:06.930 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:07.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:35:07.318 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:07.600 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:07.601 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:07.601 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:35:07.601 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:07.601 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:07.601 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:07.748 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:07.876 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install libcephfs2 2024-03-25T18:35:07.906 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:07.906 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T18:35:07.906 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:35:07.906 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T18:35:07.906 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:07.906 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:35:07.906 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:35:07.906 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-03-25T18:35:07.907 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:07.908 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 6.8 M 2024-03-25T18:35:07.908 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 33 M 2024-03-25T18:35:07.908 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:08.452 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:08.551 INFO:teuthology.orchestra.run.smithi018.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:08.607 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:08.609 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:35:08.609 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:08.690 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install libcephfs-devel 2024-03-25T18:35:08.720 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T18:35:08.745 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 820 kB/s | 248 kB 00:00 2024-03-25T18:35:08.920 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 329 kB/s | 58 kB 00:00 2024-03-25T18:35:08.995 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 681 kB/s | 51 kB 00:00 2024-03-25T18:35:09.062 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-25T18:35:09.261 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:09.338 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 154 kB/s | 138 kB 00:00 2024-03-25T18:35:09.339 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:09.340 INFO:teuthology.orchestra.run.smithi053.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-03-25T18:35:09.354 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:09.365 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:09.366 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:09.415 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:09.416 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:09.416 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T18:35:09.416 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:09.416 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:35:09.416 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:35:09.417 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:09.417 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:35:09.417 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:09.417 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:35:09.417 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:09.417 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 31 k 2024-03-25T18:35:09.417 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 140 k 2024-03-25T18:35:09.417 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:35:09.462 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:09.462 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:09.761 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:09.935 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:35:09.982 INFO:teuthology.orchestra.run.smithi018.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 55 kB/s | 31 kB 00:00 2024-03-25T18:35:09.983 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:09.983 INFO:teuthology.orchestra.run.smithi018.stdout:Total 55 kB/s | 31 kB 00:00 2024-03-25T18:35:09.983 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:35:09.987 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:35:09.987 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:35:09.994 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:35:10.019 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:35:10.020 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:35:10.164 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:35:10.292 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:10.550 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:35:10.644 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:35:10.654 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:10.713 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:35:10.743 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:10.935 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:10.935 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:10.935 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:35:10.935 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:10.935 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:10.935 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:11.100 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados2 2024-03-25T18:35:11.556 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:11.556 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:35:11.556 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:35:11.556 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:35:11.556 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:35:11.556 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:35:11.684 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:11.786 INFO:teuthology.orchestra.run.smithi018.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:11.841 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:11.843 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:35:11.843 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:11.865 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:35:11.865 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:11.865 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:11.865 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:11.865 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:35:11.866 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:35:11.866 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:35:11.866 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:35:11.866 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:35:11.866 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:11.866 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:11.900 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librbd1 2024-03-25T18:35:12.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-fuse 2024-03-25T18:35:12.480 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:12.580 INFO:teuthology.orchestra.run.smithi018.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:12.637 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:12.638 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:35:12.638 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:12.685 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:12.696 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rados 2024-03-25T18:35:12.841 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:12.841 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:12.841 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:35:12.841 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:12.841 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:12.842 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:35:12.842 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:35:12.842 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:35:12.842 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:12.842 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:12.842 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:12.842 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2024-03-25T18:35:12.842 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:12.843 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 964 k 2024-03-25T18:35:12.843 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 2.8 M 2024-03-25T18:35:12.843 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:12.980 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 572 kB/s | 79 kB 00:00 2024-03-25T18:35:13.266 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:13.365 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:13.419 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:13.420 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:35:13.420 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:13.484 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rgw 2024-03-25T18:35:14.056 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:14.155 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:14.210 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:14.211 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:35:14.211 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:14.257 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-cephfs 2024-03-25T18:35:14.827 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:14.914 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 427 kB/s | 885 kB 00:02 2024-03-25T18:35:14.915 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:14.916 INFO:teuthology.orchestra.run.smithi053.stdout:Total 465 kB/s | 964 kB 00:02 2024-03-25T18:35:14.916 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:14.926 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:14.941 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:14.941 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:14.980 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:14.981 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:35:14.982 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:15.022 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:15.023 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:15.033 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rbd 2024-03-25T18:35:15.218 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:15.360 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:35:15.441 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:15.605 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:15.703 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:15.758 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:15.760 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T18:35:15.760 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:15.809 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-fuse 2024-03-25T18:35:16.139 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:16.139 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:35:16.381 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:50 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:16.487 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:35:16.487 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:16.487 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:16.487 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:35:16.487 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:16.487 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:16.536 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:16.536 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 85 k 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 231 k 2024-03-25T18:35:16.537 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:35:16.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-volume 2024-03-25T18:35:17.127 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 145 kB/s | 85 kB 00:00 2024-03-25T18:35:17.127 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:17.128 INFO:teuthology.orchestra.run.smithi018.stdout:Total 144 kB/s | 85 kB 00:00 2024-03-25T18:35:17.128 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:35:17.136 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:35:17.136 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:35:17.195 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:35:17.195 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:35:17.338 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:17.341 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:35:17.453 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:17.495 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:17.496 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:17.496 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-03-25T18:35:17.496 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:17.496 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:17.496 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:35:17.497 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:35:17.497 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:35:17.497 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:17.497 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:17.497 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:17.497 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2024-03-25T18:35:17.497 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:17.498 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 332 k 2024-03-25T18:35:17.498 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.5 M 2024-03-25T18:35:17.498 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:17.654 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 442 kB/s | 69 kB 00:00 2024-03-25T18:35:17.852 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:18.146 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:18.146 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:18.146 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:35:18.147 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:18.147 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:18.147 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:18.403 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-mirror 2024-03-25T18:35:18.978 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:19.133 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:19.134 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:19.134 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T18:35:19.134 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:19.134 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:35:19.134 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:35:19.134 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:19.134 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:35:19.135 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:19.135 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:35:19.135 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:19.135 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 3.1 M 2024-03-25T18:35:19.135 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 13 M 2024-03-25T18:35:19.135 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:35:19.570 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 127 kB/s | 264 kB 00:02 2024-03-25T18:35:19.571 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:19.572 INFO:teuthology.orchestra.run.smithi053.stdout:Total 160 kB/s | 332 kB 00:02 2024-03-25T18:35:19.572 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:19.583 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:19.583 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:19.629 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:19.629 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:19.787 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:19.970 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:35:20.005 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:20.022 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:20.022 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:20.022 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:35:20.022 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:35:21.064 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:35:21.064 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:21.065 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:21.065 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:21.065 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:35:21.065 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:21.065 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:21.376 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados-devel 2024-03-25T18:35:21.957 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:04:56 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:22.114 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:22.115 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:22.115 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T18:35:22.115 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:22.115 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:22.115 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:35:22.116 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:22.116 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:22.116 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:22.116 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T18:35:22.116 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:22.116 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 127 k 2024-03-25T18:35:22.116 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 456 k 2024-03-25T18:35:22.116 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:22.908 INFO:teuthology.orchestra.run.smithi053.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 160 kB/s | 127 kB 00:00 2024-03-25T18:35:22.908 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:22.909 INFO:teuthology.orchestra.run.smithi053.stdout:Total 160 kB/s | 127 kB 00:00 2024-03-25T18:35:22.909 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:22.916 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:22.916 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:22.980 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:22.980 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:23.135 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:23.252 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:23.789 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:24.097 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:24.097 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:24.098 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:24.098 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:24.098 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:24.098 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:24.247 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 619 kB/s | 3.1 MB 00:05 2024-03-25T18:35:24.247 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:24.248 INFO:teuthology.orchestra.run.smithi018.stdout:Total 618 kB/s | 3.1 MB 00:05 2024-03-25T18:35:24.248 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:35:24.260 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:35:24.260 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:35:24.319 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs2 2024-03-25T18:35:24.340 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:35:24.340 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:35:24.791 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:35:24.855 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:24.879 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:24.879 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:24.880 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:35:24.880 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:24.880 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:24.880 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:24.898 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:24.998 INFO:teuthology.orchestra.run.smithi053.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:25.054 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:25.055 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:35:25.055 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:25.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs-devel 2024-03-25T18:35:25.698 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:00 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:25.853 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:25.854 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:25.854 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T18:35:25.854 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:25.854 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:25.854 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:35:25.854 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:25.854 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:25.854 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:25.855 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T18:35:25.855 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:25.855 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 31 k 2024-03-25T18:35:25.855 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 140 k 2024-03-25T18:35:25.855 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:26.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:26.064 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:26.064 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:35:26.064 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:26.064 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:26.064 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:26.344 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-nbd 2024-03-25T18:35:26.919 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:05:00 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:27.071 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:27.072 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:27.072 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:27.072 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:27.072 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 171 k 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 494 k 2024-03-25T18:35:27.073 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:35:27.859 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 218 kB/s | 171 kB 00:00 2024-03-25T18:35:27.859 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:27.859 INFO:teuthology.orchestra.run.smithi018.stdout:Total 218 kB/s | 171 kB 00:00 2024-03-25T18:35:27.860 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:35:27.869 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:35:27.869 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:35:27.931 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:35:27.931 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:35:28.084 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:35:28.184 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:28.605 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:28.888 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:28.889 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:28.889 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:35:28.889 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:28.889 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:28.889 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:35:29.126 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:35:29.701 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:35:29.860 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:35:29.860 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:29.860 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T18:35:29.860 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:29.860 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi018.stdout:Install 2 Packages 2024-03-25T18:35:29.861 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:35:29.862 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 91 M 2024-03-25T18:35:29.862 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 498 M 2024-03-25T18:35:29.862 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:35:32.753 INFO:teuthology.orchestra.run.smithi018.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 7.8 MB/s | 22 MB 00:02 2024-03-25T18:35:33.154 INFO:teuthology.orchestra.run.smithi053.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 4.3 kB/s | 31 kB 00:07 2024-03-25T18:35:33.154 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:33.154 INFO:teuthology.orchestra.run.smithi053.stdout:Total 4.3 kB/s | 31 kB 00:07 2024-03-25T18:35:33.155 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:33.157 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:33.158 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:33.200 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:33.201 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:33.325 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:33.445 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:33.785 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:34.051 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:34.051 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:34.051 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:34.051 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:34.051 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:34.051 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:34.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados2 2024-03-25T18:35:34.783 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:34.885 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:34.941 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:34.942 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:35:34.942 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:34.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librbd1 2024-03-25T18:35:35.587 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:35.686 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:35.741 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:35.743 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:35:35.743 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:35.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rados 2024-03-25T18:35:36.372 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:36.473 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:36.527 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:36.528 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:35:36.529 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:36.578 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rgw 2024-03-25T18:35:37.149 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:37.248 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:37.302 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:37.304 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:35:37.304 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:37.354 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-cephfs 2024-03-25T18:35:37.927 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:38.026 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:38.080 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:38.082 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:35:38.082 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:38.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rbd 2024-03-25T18:35:38.705 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:38.803 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:38.857 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:38.859 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T18:35:38.859 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:38.919 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-fuse 2024-03-25T18:35:39.489 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:39.644 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:39.644 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:39.644 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:39.644 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:39.644 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:39.645 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:35:39.645 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:39.645 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:39.645 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:39.645 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T18:35:39.645 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:39.646 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 85 k 2024-03-25T18:35:39.646 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 231 k 2024-03-25T18:35:39.646 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:40.331 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 124 kB/s | 85 kB 00:00 2024-03-25T18:35:40.332 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:40.332 INFO:teuthology.orchestra.run.smithi053.stdout:Total 124 kB/s | 85 kB 00:00 2024-03-25T18:35:40.332 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:40.340 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:40.340 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:40.400 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:40.401 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:40.544 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:40.648 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:41.080 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:41.378 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:41.378 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:41.378 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:41.378 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:41.378 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:41.378 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:41.631 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror 2024-03-25T18:35:42.208 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:42.364 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:42.364 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:42.364 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.1 M 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 13 M 2024-03-25T18:35:42.365 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:49.391 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 450 kB/s | 3.1 MB 00:07 2024-03-25T18:35:49.392 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:49.392 INFO:teuthology.orchestra.run.smithi053.stdout:Total 450 kB/s | 3.1 MB 00:07 2024-03-25T18:35:49.393 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:49.405 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:49.405 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:49.486 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:49.487 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:49.940 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:50.000 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:50.025 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:50.025 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:50.026 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:35:50.026 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:50.026 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:35:50.026 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:51.319 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:51.319 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:51.319 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:51.319 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:51.319 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:51.319 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:51.560 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-nbd 2024-03-25T18:35:52.136 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:27 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:52.291 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:52.291 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:52.291 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:52.291 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:52.291 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:52.291 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:35:52.291 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:52.292 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:52.292 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:52.292 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T18:35:52.292 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:52.292 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 171 k 2024-03-25T18:35:52.292 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 494 k 2024-03-25T18:35:52.292 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:35:53.552 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 136 kB/s | 171 kB 00:01 2024-03-25T18:35:53.553 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:53.553 INFO:teuthology.orchestra.run.smithi053.stdout:Total 136 kB/s | 171 kB 00:01 2024-03-25T18:35:53.553 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:35:53.563 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:35:53.564 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:35:53.626 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:35:53.627 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:35:53.890 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:35:54.011 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:54.462 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:54.756 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:54.757 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:54.757 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:35:54.757 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:54.757 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:54.757 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:35:55.020 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:35:55.594 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:05:30 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:35:55.750 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:35:55.750 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:55.750 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2024-03-25T18:35:55.751 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:35:55.752 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 91 M 2024-03-25T18:35:55.752 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 498 M 2024-03-25T18:35:55.752 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:36:41.825 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 499 kB/s | 22 MB 00:46 2024-03-25T18:38:05.115 INFO:teuthology.orchestra.run.smithi018.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 449 kB/s | 68 MB 02:35 2024-03-25T18:38:05.116 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:05.116 INFO:teuthology.orchestra.run.smithi018.stdout:Total 597 kB/s | 91 MB 02:35 2024-03-25T18:38:05.116 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:38:05.120 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:38:05.120 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:38:05.744 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:38:05.744 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:38:09.015 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:38:16.531 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:38:16.608 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:20.020 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:20.021 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:38:20.296 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:38:20.296 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:38:20.296 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:38:20.296 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:20.296 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:20.296 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:38:20.296 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:38:20.521 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:38:21.094 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:07:55 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:38:21.249 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:38:21.249 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:38:21.249 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T18:38:21.249 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:38:21.249 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:38:21.249 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:38:21.249 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:38:21.249 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:38:21.250 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:38:21.250 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:38:21.250 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:38:21.250 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 416 M 2024-03-25T18:38:21.250 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.7 G 2024-03-25T18:38:21.250 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:39:14.675 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 351 kB/s | 68 MB 03:18 2024-03-25T18:39:14.676 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:39:14.676 INFO:teuthology.orchestra.run.smithi053.stdout:Total 466 kB/s | 91 MB 03:18 2024-03-25T18:39:14.676 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:39:14.680 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:39:14.680 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:39:15.292 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:39:15.292 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:39:18.601 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:39:26.123 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:39:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:29.509 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:29.509 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:39:29.781 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:39:29.781 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:39:29.781 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T18:39:29.781 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:29.781 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:29.781 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:39:29.781 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T18:39:30.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:39:30.718 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:09:05 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T18:39:30.877 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T18:39:30.877 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:39:30.877 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 416 M 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.7 G 2024-03-25T18:39:30.878 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T18:51:29.646 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 540 kB/s | 416 MB 13:08 2024-03-25T18:51:29.646 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:29.646 INFO:teuthology.orchestra.run.smithi018.stdout:Total 540 kB/s | 416 MB 13:08 2024-03-25T18:51:29.647 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:51:29.650 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:51:29.650 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:51:32.238 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:51:32.238 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:52:16.853 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:52:16.948 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:52:23.880 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:52:24.139 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:52:24.139 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:52:24.139 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:52:24.139 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:24.139 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:52:24.140 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:52:24.403 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:52:24.977 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:21:58 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:52:25.131 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:52:25.132 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:52:25.132 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T18:52:25.132 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:52:25.132 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 1.4 M 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 6.4 M 2024-03-25T18:52:25.133 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:52:28.383 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 450 kB/s | 1.4 MB 00:03 2024-03-25T18:52:28.384 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:28.384 INFO:teuthology.orchestra.run.smithi018.stdout:Total 449 kB/s | 1.4 MB 00:03 2024-03-25T18:52:28.384 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:52:28.388 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:52:28.388 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T18:52:28.435 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T18:52:28.435 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T18:52:28.700 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T18:52:28.800 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:29.187 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:29.514 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:52:29.514 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:52:29.515 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T18:52:29.515 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:29.515 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:52:29.515 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T18:52:29.698 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:52:30.268 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:22:04 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T18:52:30.423 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 176 M 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 753 M 2024-03-25T18:52:30.424 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T18:59:10.298 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 361 kB/s | 416 MB 19:39 2024-03-25T18:59:10.299 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:10.299 INFO:teuthology.orchestra.run.smithi053.stdout:Total 361 kB/s | 416 MB 19:39 2024-03-25T18:59:10.299 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T18:59:10.302 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T18:59:10.302 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T18:59:12.890 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T18:59:12.891 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T18:59:57.901 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T18:59:58.013 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:59:59.444 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 401 kB/s | 176 MB 07:29 2024-03-25T18:59:59.444 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:59.445 INFO:teuthology.orchestra.run.smithi018.stdout:Total 401 kB/s | 176 MB 07:29 2024-03-25T18:59:59.445 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T18:59:59.449 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T18:59:59.449 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:00:00.559 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:00:00.559 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:00:04.986 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:00:05.241 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:00:05.241 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:00:05.241 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:00:05.241 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:05.241 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:00:05.242 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:00:05.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T19:00:06.092 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:29:41 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:00:06.245 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2024-03-25T19:00:06.249 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:00:06.250 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:00:06.250 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 1.4 M 2024-03-25T19:00:06.250 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 6.4 M 2024-03-25T19:00:06.250 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:00:09.937 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 396 kB/s | 1.4 MB 00:03 2024-03-25T19:00:09.938 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:00:09.938 INFO:teuthology.orchestra.run.smithi053.stdout:Total 396 kB/s | 1.4 MB 00:03 2024-03-25T19:00:09.938 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:00:09.943 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:00:09.943 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:00:09.991 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:00:09.991 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:00:10.260 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:00:10.373 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:00:10.773 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:00:11.127 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:00:11.127 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:00:11.127 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:00:11.127 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:11.127 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:00:11.127 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:00:11.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T19:00:11.904 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:29:46 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:00:12.059 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:00:12.060 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:00:12.060 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:00:12.060 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:00:12.060 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:00:12.060 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T19:00:12.060 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:00:12.060 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:00:12.060 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:00:12.061 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:00:12.061 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:00:12.061 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 176 M 2024-03-25T19:00:12.061 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 753 M 2024-03-25T19:00:12.061 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:00:19.635 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:00:19.726 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:00:24.353 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:00:24.634 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:00:24.634 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:00:24.634 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:00:24.634 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:24.634 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:00:24.634 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:00:24.857 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:00:25.430 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:29:59 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:00:25.583 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:00:25.584 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:00:25.584 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:00:25.584 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:00:25.584 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:00:25.584 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:00:25.585 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:00:25.585 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:00:25.585 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:00:25.585 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:00:25.585 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:00:25.585 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 672 M 2024-03-25T19:00:25.585 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 3.1 G 2024-03-25T19:00:25.585 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:07:27.526 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 413 kB/s | 176 MB 07:15 2024-03-25T19:07:27.527 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:27.527 INFO:teuthology.orchestra.run.smithi053.stdout:Total 413 kB/s | 176 MB 07:15 2024-03-25T19:07:27.527 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:07:27.529 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:07:27.530 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:07:28.638 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:07:28.639 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:07:47.760 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:07:47.859 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:07:52.480 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:07:52.764 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:07:52.764 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:07:52.764 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:07:52.764 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:52.764 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:07:52.764 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:07:53.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:07:53.587 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:37:28 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:07:53.742 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:07:53.742 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:07:53.742 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:07:53.742 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:07:53.742 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:07:53.743 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:07:53.743 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:07:53.743 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:07:53.743 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:07:53.743 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:07:53.743 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:07:53.743 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 672 M 2024-03-25T19:07:53.743 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 3.1 G 2024-03-25T19:07:53.744 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:21:28.589 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 545 kB/s | 672 MB 21:03 2024-03-25T19:21:28.590 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:28.590 INFO:teuthology.orchestra.run.smithi018.stdout:Total 545 kB/s | 672 MB 21:03 2024-03-25T19:21:28.590 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:21:28.594 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:21:28.594 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:21:32.752 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:21:32.752 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:22:47.982 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:22:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:56.963 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:57.232 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:22:57.232 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:22:57.232 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:22:57.232 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:57.232 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:22:57.233 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:22:57.550 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:22:58.125 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:52:32 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:22:58.225 INFO:teuthology.orchestra.run.smithi018.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:22:58.280 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:22:58.282 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T19:22:58.282 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:22:58.476 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:22:59.067 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:52:33 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:22:59.223 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:22:59.224 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:22:59.225 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 20 M 2024-03-25T19:22:59.225 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 103 M 2024-03-25T19:22:59.225 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:23:42.852 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 462 kB/s | 20 MB 00:43 2024-03-25T19:23:42.852 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:42.852 INFO:teuthology.orchestra.run.smithi018.stdout:Total 462 kB/s | 20 MB 00:43 2024-03-25T19:23:42.852 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:23:42.856 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:23:42.857 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:23:43.013 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:23:43.014 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:23:45.369 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:23:45.486 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:46.406 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:46.684 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:46.684 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:23:46.684 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:23:46.684 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:46.684 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:23:46.685 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:23:46.876 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:23:47.446 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:53:21 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:23:47.600 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 48 M 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 173 M 2024-03-25T19:23:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:24:46.329 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 832 kB/s | 48 MB 00:58 2024-03-25T19:24:46.330 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:46.330 INFO:teuthology.orchestra.run.smithi018.stdout:Total 832 kB/s | 48 MB 00:58 2024-03-25T19:24:46.330 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:24:46.334 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:24:46.334 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:24:46.661 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:24:46.662 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:24:51.781 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:24:51.879 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:53.383 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:53.655 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:53.655 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:24:53.655 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:24:53.655 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:53.655 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:24:53.655 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:24:53.852 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:24:54.424 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:54:28 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:24:54.578 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:24:54.579 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:24:54.580 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 99 M 2024-03-25T19:24:54.580 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 449 M 2024-03-25T19:24:54.580 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:27:20.160 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 693 kB/s | 99 MB 02:25 2024-03-25T19:27:20.161 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:20.161 INFO:teuthology.orchestra.run.smithi018.stdout:Total 693 kB/s | 99 MB 02:25 2024-03-25T19:27:20.161 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:27:20.163 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:27:20.164 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:27:20.802 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:27:20.802 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:27:31.685 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:27:31.792 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:34.775 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:35.061 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:35.061 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:27:35.061 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:27:35.061 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:35.061 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:27:35.061 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:27:35.237 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:27:35.809 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:57:09 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:27:35.966 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 287 M 2024-03-25T19:27:35.967 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.2 G 2024-03-25T19:27:35.968 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:29:39.372 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 527 kB/s | 672 MB 21:45 2024-03-25T19:29:39.372 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:39.372 INFO:teuthology.orchestra.run.smithi053.stdout:Total 527 kB/s | 672 MB 21:45 2024-03-25T19:29:39.372 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:29:39.376 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:29:39.376 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:29:43.532 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:29:43.532 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:30:58.573 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:30:59.142 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:06.915 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:07.194 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:07.194 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:31:07.194 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:31:07.194 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:07.194 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:31:07.194 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:31:07.485 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:31:08.057 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:00:43 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:31:08.157 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:31:08.213 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:31:08.214 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-03-25T19:31:08.214 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:31:08.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:31:08.866 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:00:43 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:31:09.021 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:31:09.021 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:31:09.022 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:31:09.023 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 20 M 2024-03-25T19:31:09.023 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 103 M 2024-03-25T19:31:09.023 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:31:32.564 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 857 kB/s | 20 MB 00:23 2024-03-25T19:31:32.565 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:32.566 INFO:teuthology.orchestra.run.smithi053.stdout:Total 857 kB/s | 20 MB 00:23 2024-03-25T19:31:32.566 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:31:32.570 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:31:32.570 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:31:32.727 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:31:32.727 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:31:35.090 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:31:35.199 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:36.467 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:36.467 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:31:36.467 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:31:36.467 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:36.467 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:31:36.467 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:31:36.672 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:31:37.246 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:01:12 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:31:37.401 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:31:37.401 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:31:37.401 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:31:37.401 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 48 M 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 173 M 2024-03-25T19:31:37.402 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:32:04.764 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.7 MB/s | 48 MB 00:27 2024-03-25T19:32:04.765 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:04.765 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.7 MB/s | 48 MB 00:27 2024-03-25T19:32:04.765 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:32:04.769 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:32:04.769 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:32:05.099 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:32:05.099 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:32:10.116 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:32:10.198 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:11.515 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:11.785 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:11.786 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:32:11.786 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:32:11.786 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:11.786 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:32:11.786 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:32:12.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:32:12.585 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:01:47 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:32:12.740 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:32:12.741 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:32:12.741 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:32:12.741 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:32:12.741 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:32:12.741 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:32:12.741 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:32:12.742 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:32:12.742 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:32:12.742 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:32:12.742 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:32:12.742 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 99 M 2024-03-25T19:32:12.742 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 449 M 2024-03-25T19:32:12.742 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:33:03.121 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 898 kB/s | 287 MB 05:27 2024-03-25T19:33:03.122 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:03.122 INFO:teuthology.orchestra.run.smithi018.stdout:Total 898 kB/s | 287 MB 05:27 2024-03-25T19:33:03.122 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:33:03.126 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:33:03.126 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:33:04.920 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:33:04.921 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:33:36.019 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:33:36.151 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:43.624 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:43.905 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:43.905 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:43.905 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:33:43.905 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:43.905 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:43.905 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:33:44.146 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:33:44.719 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:03:18 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:33:44.873 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:33:44.873 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:33:44.874 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:44.875 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 5.9 M 2024-03-25T19:33:44.875 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 28 M 2024-03-25T19:33:44.875 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:33:51.285 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 949 kB/s | 5.9 MB 00:06 2024-03-25T19:33:51.286 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:51.286 INFO:teuthology.orchestra.run.smithi018.stdout:Total 949 kB/s | 5.9 MB 00:06 2024-03-25T19:33:51.286 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:33:51.291 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:33:51.291 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:33:51.365 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:33:51.366 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:33:51.814 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 99 MB 01:39 2024-03-25T19:33:51.814 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:51.815 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.0 MB/s | 99 MB 01:39 2024-03-25T19:33:51.815 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:33:51.818 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:33:51.818 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:33:52.138 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:33:52.237 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:33:52.456 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:33:52.852 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:33:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:33:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:53.131 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:33:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:53.131 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:53.131 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:33:53.316 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:33:53.890 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:03:27 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:33:54.049 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:33:54.050 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:54.050 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:33:54.050 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:54.050 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:33:54.050 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:33:54.050 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:54.050 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:33:54.050 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:54.051 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:33:54.051 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:54.051 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 109 k 2024-03-25T19:33:54.051 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 440 k 2024-03-25T19:33:54.051 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:33:54.605 INFO:teuthology.orchestra.run.smithi018.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 196 kB/s | 109 kB 00:00 2024-03-25T19:33:54.605 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:54.605 INFO:teuthology.orchestra.run.smithi018.stdout:Total 196 kB/s | 109 kB 00:00 2024-03-25T19:33:54.605 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:33:54.609 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:33:54.609 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:33:54.647 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:33:54.647 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:33:54.773 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:33:55.007 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:55.366 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:55.647 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:33:55.647 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:55.647 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:33:55.647 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:55.647 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:55.647 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:33:55.847 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:33:56.419 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:03:30 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:33:56.574 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:33:56.574 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:56.574 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:33:56.574 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:56.574 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:33:56.574 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:33:56.575 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:56.575 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:33:56.575 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:33:56.575 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:33:56.575 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:33:56.575 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 5.4 M 2024-03-25T19:33:56.575 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 25 M 2024-03-25T19:33:56.575 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:34:02.007 INFO:teuthology.orchestra.run.smithi018.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 5.4 MB 00:05 2024-03-25T19:34:02.008 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:02.008 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.0 MB/s | 5.4 MB 00:05 2024-03-25T19:34:02.008 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:34:02.012 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:34:02.012 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:34:02.084 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:34:02.084 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:34:02.785 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:34:02.894 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:03.325 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:34:03.415 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:03.563 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:03.854 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:34:03.854 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:34:03.854 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:34:03.854 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:03.854 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:34:03.854 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:34:04.168 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados2-debuginfo 2024-03-25T19:34:04.743 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:03:38 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:34:04.898 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:34:04.898 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 44 M 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 253 M 2024-03-25T19:34:04.899 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:34:06.241 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:06.520 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:06.520 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:34:06.520 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:34:06.520 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:06.520 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:34:06.520 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:34:06.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:34:07.295 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:03:42 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:34:07.448 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:34:07.448 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:34:07.448 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:34:07.449 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 287 M 2024-03-25T19:34:07.450 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.2 G 2024-03-25T19:34:07.450 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:34:37.505 INFO:teuthology.orchestra.run.smithi018.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.4 MB/s | 44 MB 00:32 2024-03-25T19:34:37.506 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:37.506 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.4 MB/s | 44 MB 00:32 2024-03-25T19:34:37.507 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:34:37.511 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:34:37.511 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:34:37.826 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:34:37.826 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:34:43.155 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:34:43.265 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:34:45.206 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:34:45.490 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:34:45.490 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:34:45.490 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:34:45.490 INFO:teuthology.orchestra.run.smithi018.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:45.490 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:34:45.490 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:34:45.686 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:34:46.259 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:04:20 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:34:46.413 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:34:46.414 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:34:46.415 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:34:46.415 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 52 M 2024-03-25T19:34:46.415 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 308 M 2024-03-25T19:34:46.415 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:34:56.940 INFO:teuthology.orchestra.run.smithi018.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 4.9 MB/s | 52 MB 00:10 2024-03-25T19:34:56.941 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:56.941 INFO:teuthology.orchestra.run.smithi018.stdout:Total 4.9 MB/s | 52 MB 00:10 2024-03-25T19:34:56.941 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:34:56.945 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:34:56.946 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:34:57.307 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:34:57.307 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:35:03.663 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:35:03.757 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:35:05.924 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:35:06.200 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:35:06.200 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:06.200 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:35:06.200 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:06.200 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:06.201 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:35:06.402 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:35:06.971 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:04:40 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:35:07.127 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 615 k 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.1 M 2024-03-25T19:35:07.128 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:35:08.034 INFO:teuthology.orchestra.run.smithi018.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 679 kB/s | 615 kB 00:00 2024-03-25T19:35:08.035 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:08.035 INFO:teuthology.orchestra.run.smithi018.stdout:Total 678 kB/s | 615 kB 00:00 2024-03-25T19:35:08.035 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:35:08.039 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:35:08.039 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:35:08.081 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:35:08.081 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:35:08.256 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:35:08.352 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:08.768 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:09.051 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:09.051 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:09.051 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:35:09.051 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:09.051 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:09.051 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:35:09.245 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:35:09.816 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:04:43 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:35:09.975 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:35:09.975 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:09.975 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:35:09.975 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:09.975 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:35:09.976 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:35:09.976 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:09.976 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:35:09.976 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:09.976 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:35:09.976 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:09.976 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 771 k 2024-03-25T19:35:09.976 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.7 M 2024-03-25T19:35:09.977 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:35:11.004 INFO:teuthology.orchestra.run.smithi018.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 750 kB/s | 771 kB 00:01 2024-03-25T19:35:11.005 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:11.005 INFO:teuthology.orchestra.run.smithi018.stdout:Total 749 kB/s | 771 kB 00:01 2024-03-25T19:35:11.005 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:35:11.009 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:35:11.009 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:35:11.051 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:35:11.051 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:35:11.237 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:35:11.347 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:35:11.728 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:35:12.008 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:35:12.008 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:12.008 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:35:12.008 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:12.008 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:12.008 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:35:12.202 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:35:12.771 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:04:46 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:35:12.926 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:35:12.926 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:12.926 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:35:12.926 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:12.926 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 1.1 M 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 4.3 M 2024-03-25T19:35:12.927 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:35:14.316 INFO:teuthology.orchestra.run.smithi018.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 829 kB/s | 1.1 MB 00:01 2024-03-25T19:35:14.316 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:14.317 INFO:teuthology.orchestra.run.smithi018.stdout:Total 828 kB/s | 1.1 MB 00:01 2024-03-25T19:35:14.317 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:35:14.321 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:35:14.321 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:35:14.367 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:35:14.367 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:35:14.595 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:35:14.692 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:35:15.089 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:35:15.356 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:35:15.356 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:15.356 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:35:15.357 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:15.357 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:15.357 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:35:15.563 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:35:16.134 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:04:50 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:35:16.289 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:35:16.289 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:16.289 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:35:16.289 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:16.289 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:35:16.289 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:35:16.289 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:16.290 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:35:16.290 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:16.290 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:35:16.290 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:16.290 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 337 k 2024-03-25T19:35:16.290 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.0 M 2024-03-25T19:35:16.290 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:35:16.919 INFO:teuthology.orchestra.run.smithi018.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 537 kB/s | 337 kB 00:00 2024-03-25T19:35:16.919 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:16.920 INFO:teuthology.orchestra.run.smithi018.stdout:Total 536 kB/s | 337 kB 00:00 2024-03-25T19:35:16.920 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:35:16.924 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:35:16.924 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:35:16.963 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:35:16.963 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:35:17.107 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:35:17.221 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:35:17.620 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:35:17.899 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:35:17.899 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:17.899 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:35:17.900 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:17.900 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:17.900 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:35:18.085 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:35:18.656 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:04:52 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:18.812 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:35:18.813 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:18.813 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 509 k 2024-03-25T19:35:18.813 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.5 M 2024-03-25T19:35:18.813 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:35:19.914 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 463 kB/s | 509 kB 00:01 2024-03-25T19:35:19.914 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:19.915 INFO:teuthology.orchestra.run.smithi018.stdout:Total 462 kB/s | 509 kB 00:01 2024-03-25T19:35:19.915 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:35:19.919 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:35:19.919 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:35:19.962 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:35:19.962 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:35:20.132 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:35:20.233 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:20.635 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:20.932 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:20.932 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:20.932 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:35:20.932 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:20.932 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:20.932 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:35:21.111 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:35:21.686 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:04:55 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:35:21.842 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:35:21.843 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:35:21.844 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 52 M 2024-03-25T19:35:21.844 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 298 M 2024-03-25T19:35:21.844 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:36:29.608 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 779 kB/s | 52 MB 01:07 2024-03-25T19:36:29.609 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:29.609 INFO:teuthology.orchestra.run.smithi018.stdout:Total 779 kB/s | 52 MB 01:07 2024-03-25T19:36:29.609 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:36:29.612 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:36:29.612 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:36:29.963 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:36:29.963 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:36:36.234 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:36:36.366 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:38.409 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:38.686 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:38.686 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:36:38.686 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:36:38.687 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:38.687 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:36:38.687 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:36:38.871 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:36:39.442 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:06:13 ago on Mon 25 Mar 2024 06:30:26 PM UTC. 2024-03-25T19:36:39.596 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T19:36:39.597 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:36:39.597 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T19:36:39.597 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:36:39.597 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 1.3 M 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 5.5 M 2024-03-25T19:36:39.598 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T19:36:41.194 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 803 kB/s | 1.3 MB 00:01 2024-03-25T19:36:41.195 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:41.196 INFO:teuthology.orchestra.run.smithi018.stdout:Total 802 kB/s | 1.3 MB 00:01 2024-03-25T19:36:41.196 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T19:36:41.200 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T19:36:41.200 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T19:36:41.248 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T19:36:41.248 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T19:36:41.503 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T19:36:41.621 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:36:42.037 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:36:42.377 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:36:42.377 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:36:42.377 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T19:36:42.377 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:42.378 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:36:42.378 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T19:36:42.570 DEBUG:teuthology.parallel:result is None 2024-03-25T19:37:52.757 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 287 MB 03:45 2024-03-25T19:37:52.758 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:52.758 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.3 MB/s | 287 MB 03:45 2024-03-25T19:37:52.758 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:37:52.762 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:37:52.762 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:37:54.552 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:37:54.552 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:38:25.776 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:38:26.316 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:27.823 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:28.144 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:28.144 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:28.144 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:38:28.144 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:28.144 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:28.144 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:38:28.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:38:28.969 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:08:03 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:38:29.123 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:38:29.124 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:29.125 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 5.9 M 2024-03-25T19:38:29.125 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 28 M 2024-03-25T19:38:29.125 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:38:32.724 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.7 MB/s | 5.9 MB 00:03 2024-03-25T19:38:32.725 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:32.725 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.7 MB/s | 5.9 MB 00:03 2024-03-25T19:38:32.725 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:38:32.730 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:38:32.730 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:38:32.805 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:38:32.805 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:38:33.574 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:38:33.675 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:34.241 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:34.517 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:34.517 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:38:34.517 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:34.518 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:38:34.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:38:35.313 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:08:10 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:38:35.470 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:38:35.471 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:35.472 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 109 k 2024-03-25T19:38:35.472 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 440 k 2024-03-25T19:38:35.472 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:38:36.010 INFO:teuthology.orchestra.run.smithi053.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 202 kB/s | 109 kB 00:00 2024-03-25T19:38:36.010 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:36.011 INFO:teuthology.orchestra.run.smithi053.stdout:Total 201 kB/s | 109 kB 00:00 2024-03-25T19:38:36.011 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:38:36.015 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:38:36.015 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:38:36.052 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:38:36.052 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:38:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:38:36.295 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:36.735 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:37.010 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:37.010 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:37.010 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:38:37.010 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:37.010 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:37.010 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:38:37.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:38:37.797 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:08:12 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:38:37.951 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:38:37.952 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:37.952 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 5.4 M 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 25 M 2024-03-25T19:38:37.953 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:38:40.721 INFO:teuthology.orchestra.run.smithi053.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 5.4 MB 00:02 2024-03-25T19:38:40.722 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:40.723 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.9 MB/s | 5.4 MB 00:02 2024-03-25T19:38:40.723 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:38:40.727 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:38:40.727 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:38:40.799 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:38:40.799 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:38:41.496 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:38:41.607 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:42.128 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:42.400 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:42.400 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:42.400 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:38:42.400 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:42.400 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:42.400 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:38:42.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados2-debuginfo 2024-03-25T19:38:43.149 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:08:18 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:38:43.304 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:38:43.304 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:43.304 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 44 M 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 253 M 2024-03-25T19:38:43.305 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:38:58.156 INFO:teuthology.orchestra.run.smithi053.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 3.0 MB/s | 44 MB 00:14 2024-03-25T19:38:58.157 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:58.157 INFO:teuthology.orchestra.run.smithi053.stdout:Total 3.0 MB/s | 44 MB 00:14 2024-03-25T19:38:58.157 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:38:58.162 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:38:58.162 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:38:58.476 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:38:58.477 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:03.833 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:03.934 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:05.795 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:06.065 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:06.065 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:06.065 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:06.065 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:06.065 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:06.065 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:39:06.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:39:06.854 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:08:41 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:39:07.009 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:39:07.009 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:07.009 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 52 M 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 308 M 2024-03-25T19:39:07.010 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:39:18.458 INFO:teuthology.orchestra.run.smithi053.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 4.5 MB/s | 52 MB 00:11 2024-03-25T19:39:18.459 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:18.459 INFO:teuthology.orchestra.run.smithi053.stdout:Total 4.5 MB/s | 52 MB 00:11 2024-03-25T19:39:18.459 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:39:18.465 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:39:18.465 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:39:18.825 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:39:18.825 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:25.276 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:27.628 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:27.911 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:27.911 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:27.911 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:27.911 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:27.911 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:27.911 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:39:28.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:39:28.684 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:09:03 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:39:28.840 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:39:28.840 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:28.840 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 615 k 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 2.1 M 2024-03-25T19:39:28.841 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:39:29.611 INFO:teuthology.orchestra.run.smithi053.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 799 kB/s | 615 kB 00:00 2024-03-25T19:39:29.611 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:29.611 INFO:teuthology.orchestra.run.smithi053.stdout:Total 798 kB/s | 615 kB 00:00 2024-03-25T19:39:29.612 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:39:29.615 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:39:29.615 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:39:29.658 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:39:29.658 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:29.831 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:29.937 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:30.354 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:30.644 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:30.644 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:30.644 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:30.644 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:30.644 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:30.644 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:39:30.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:39:31.448 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:09:06 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:39:31.605 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:39:31.606 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:39:31.606 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:31.606 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:39:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:39:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:31.607 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:39:31.607 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:31.607 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:39:31.607 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:31.607 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 771 k 2024-03-25T19:39:31.607 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 2.7 M 2024-03-25T19:39:31.607 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:39:31.914 INFO:teuthology.orchestra.run.smithi053.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 2.5 MB/s | 771 kB 00:00 2024-03-25T19:39:31.914 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:31.914 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.4 MB/s | 771 kB 00:00 2024-03-25T19:39:31.915 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:39:31.918 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:39:31.918 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:39:31.957 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:39:31.957 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:32.144 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:32.250 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:32.692 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:32.979 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:32.979 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:32.979 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:32.979 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:32.979 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:32.979 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:39:33.181 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:39:33.757 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:09:08 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:39:33.913 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:39:33.913 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:33.913 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 1.1 M 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 4.3 M 2024-03-25T19:39:33.914 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:39:34.425 INFO:teuthology.orchestra.run.smithi053.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 2.2 MB/s | 1.1 MB 00:00 2024-03-25T19:39:34.426 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:34.426 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.2 MB/s | 1.1 MB 00:00 2024-03-25T19:39:34.426 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:39:34.430 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:39:34.430 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:39:34.472 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:39:34.473 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:34.702 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:34.793 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:35.213 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:35.496 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:35.496 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:35.496 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:35.496 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:35.496 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:35.496 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:39:35.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:39:36.309 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:09:11 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:39:36.465 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:39:36.465 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 337 k 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.0 M 2024-03-25T19:39:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:39:36.720 INFO:teuthology.orchestra.run.smithi053.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 1.3 MB/s | 337 kB 00:00 2024-03-25T19:39:36.721 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:36.721 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-03-25T19:39:36.721 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:39:36.725 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:39:36.725 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:39:36.762 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:39:36.762 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:36.907 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:37.022 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:37.403 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:37.669 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:37.669 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:37.669 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:37.669 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:37.669 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:37.669 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:39:37.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:39:38.466 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:09:13 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:39:38.621 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:38.622 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:39:38.623 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:38.623 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 509 k 2024-03-25T19:39:38.623 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 2.5 M 2024-03-25T19:39:38.623 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:39:39.523 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 565 kB/s | 509 kB 00:00 2024-03-25T19:39:39.524 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:39.524 INFO:teuthology.orchestra.run.smithi053.stdout:Total 564 kB/s | 509 kB 00:00 2024-03-25T19:39:39.524 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:39:39.528 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:39:39.528 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:39:39.571 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:39:39.571 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:39.768 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:39.858 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:40.293 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:40.576 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:40.577 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:40.577 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:40.577 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:40.577 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:40.577 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:39:40.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:39:41.360 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:09:16 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:39:41.515 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:39:41.516 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:41.516 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:39:41.516 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:41.516 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:39:41.516 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:41.516 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:41.517 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:39:41.517 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:41.517 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:39:41.517 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:41.517 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 52 M 2024-03-25T19:39:41.517 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 298 M 2024-03-25T19:39:41.517 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:39:47.551 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 8.5 MB/s | 52 MB 00:06 2024-03-25T19:39:47.552 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:47.552 INFO:teuthology.orchestra.run.smithi053.stdout:Total 8.5 MB/s | 52 MB 00:06 2024-03-25T19:39:47.552 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:39:47.556 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:39:47.557 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:39:47.909 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:39:47.909 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:54.176 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:54.289 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:56.361 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:56.632 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:56.632 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:56.632 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:56.632 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:56.632 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:56.632 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:39:56.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:39:57.433 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 1:09:32 ago on Mon 25 Mar 2024 06:30:25 PM UTC. 2024-03-25T19:39:57.588 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:57.589 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 1.3 M 2024-03-25T19:39:57.590 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 5.5 M 2024-03-25T19:39:57.590 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-25T19:39:58.828 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 1.3 MB 00:01 2024-03-25T19:39:58.829 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:58.829 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.0 MB/s | 1.3 MB 00:01 2024-03-25T19:39:58.829 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-25T19:39:58.833 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-25T19:39:58.833 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-25T19:39:58.882 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-25T19:39:58.882 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-25T19:39:59.149 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-25T19:39:59.246 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:59.699 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:59.983 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-25T19:39:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:59.983 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:39:59.983 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-25T19:40:00.199 DEBUG:teuthology.parallel:result is None 2024-03-25T19:40:00.200 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:40:00.415 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:40:00.439 INFO:teuthology.orchestra.run.smithi018.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:00.440 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:00.440 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:40:00.441 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:40:00.659 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:40:00.680 INFO:teuthology.orchestra.run.smithi053.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:00.680 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:00.681 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:40:00.682 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:40:00.683 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:00.683 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:40:00.714 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:00.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:40:00.748 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:40:00.748 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:00.749 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:40:00.784 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:40:00.853 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:00.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:40:00.881 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:40:00.948 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:40:00.949 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:00.949 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:40:00.977 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:40:01.046 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:01.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:40:01.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:40:01.140 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:40:01.140 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:01.140 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:40:01.169 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:40:01.237 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:01.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:40:01.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:40:01.330 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:40:01.415 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:40:01.415 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 777 /var/log/ceph 2024-03-25T19:40:01.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2024-03-25T19:40:01.444 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:40:01.444 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:40:01.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:40:01.514 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:40:01.514 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:40:01.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:40:01.594 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:40:01.594 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:40:01.594 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621323', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi018.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIf8qtBW1VRnU77oStvxOR7mFLCKGb1HTnzzU0QgDPNLlMjYNsWy8DxR7LMFeOQOQ9GDGrEIsPPh/wjTS8d9ASk=', 'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4uEslQyHioVMal//nl/QuVFFOzb5Vxx/XC+yGG8DaL9Py/goUzcz2fw0wljTg06zHW5budGXLp7J9jX4RObhg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:40:01.595 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:40:01.625 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:40:01.640 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:40:01.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:40:01.708 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:01.708 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:40:01.763 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:40:01.763 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_1 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:01.818 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:01.819 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-25 18:35:25.119453965 +0000 2024-03-25T19:40:01.819 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-25 18:29:27.191727857 +0000 2024-03-25T19:40:01.819 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-25 18:29:27.191727857 +0000 2024-03-25T19:40:01.819 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-25 18:29:27.191727857 +0000 2024-03-25T19:40:01.819 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:40:01.884 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-25T19:40:01.884 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-25T19:40:01.884 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000117992 s, 4.3 MB/s 2024-03-25T19:40:01.886 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:40:01.942 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_2 2024-03-25T19:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:01.997 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-25 18:35:25.120453953 +0000 2024-03-25T19:40:01.998 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-25 18:29:27.540723723 +0000 2024-03-25T19:40:01.998 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-25 18:29:27.540723723 +0000 2024-03-25T19:40:01.998 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-25 18:29:27.540723723 +0000 2024-03-25T19:40:01.998 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:40:02.062 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-25T19:40:02.063 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-25T19:40:02.063 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 9.869e-05 s, 5.2 MB/s 2024-03-25T19:40:02.064 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:40:02.119 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_3 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-25 18:35:25.120453953 +0000 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-25 18:29:27.876719743 +0000 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-25 18:29:27.876719743 +0000 2024-03-25T19:40:02.175 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-25 18:29:27.876719743 +0000 2024-03-25T19:40:02.176 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:40:02.240 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-25T19:40:02.240 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-25T19:40:02.241 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 9.0769e-05 s, 5.6 MB/s 2024-03-25T19:40:02.242 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:40:02.298 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_4 2024-03-25T19:40:02.352 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:40:02.352 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:02.353 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T19:40:02.353 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:02.353 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:02.353 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-25 18:35:25.120453953 +0000 2024-03-25T19:40:02.353 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-25 18:29:28.206715834 +0000 2024-03-25T19:40:02.353 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-25 18:29:28.206715834 +0000 2024-03-25T19:40:02.353 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-25 18:29:28.206715834 +0000 2024-03-25T19:40:02.353 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:40:02.417 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-25T19:40:02.418 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-25T19:40:02.418 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000110186 s, 4.6 MB/s 2024-03-25T19:40:02.419 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:40:02.475 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:02.475 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:02.475 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:40:02.492 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:40:02.492 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-25 18:35:50.252359546 +0000 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-25 18:29:27.190110780 +0000 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-25 18:29:27.191110760 +0000 2024-03-25T19:40:02.549 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-25 18:29:27.190110780 +0000 2024-03-25T19:40:02.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:40:02.617 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-25T19:40:02.618 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-25T19:40:02.618 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000109814 s, 4.7 MB/s 2024-03-25T19:40:02.619 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:40:02.677 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-03-25T19:40:02.733 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:40:02.733 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:02.733 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:40:02.733 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:02.734 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:02.734 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-25 18:35:50.252359546 +0000 2024-03-25T19:40:02.734 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-25 18:29:27.538103809 +0000 2024-03-25T19:40:02.734 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-25 18:29:27.538103809 +0000 2024-03-25T19:40:02.734 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-25 18:29:27.538103809 +0000 2024-03-25T19:40:02.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:40:02.800 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-25T19:40:02.801 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-25T19:40:02.801 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000104617 s, 4.9 MB/s 2024-03-25T19:40:02.802 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:40:02.859 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-25 18:35:50.253359526 +0000 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-25 18:29:27.863097299 +0000 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-25 18:29:27.863097299 +0000 2024-03-25T19:40:02.915 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-25 18:29:27.863097299 +0000 2024-03-25T19:40:02.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:40:02.982 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-25T19:40:02.982 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-25T19:40:02.982 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000107132 s, 4.8 MB/s 2024-03-25T19:40:02.984 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:40:03.040 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-03-25T19:40:03.095 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:40:03.095 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:03.095 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:40:03.095 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:03.095 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:03.095 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-25 18:35:50.253359526 +0000 2024-03-25T19:40:03.095 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-25 18:29:28.181090928 +0000 2024-03-25T19:40:03.095 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-25 18:29:28.181090928 +0000 2024-03-25T19:40:03.096 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-25 18:29:28.181090928 +0000 2024-03-25T19:40:03.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:40:03.161 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-25T19:40:03.161 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-25T19:40:03.161 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.9678e-05 s, 5.1 MB/s 2024-03-25T19:40:03.162 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:40:03.219 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:03.219 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:40:03.219 INFO:tasks.ceph:Generating config... 2024-03-25T19:40:03.220 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:40:03.220 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:40:03.220 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:40:03.220 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:40:03.220 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:40:03.220 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:40:03.220 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:40:03.220 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:40:03.220 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:40:03.220 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:40:03.221 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:40:03.221 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:40:03.221 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:40:03.221 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:40:03.221 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:40:03.222 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:40:03.223 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:40:03.223 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:40:03.223 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:03.283 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:40:03.286 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:40:03.333 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:03.361 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.18:6789]'), ('mon.c', '[v1:172.21.15.18:6790]'), ('mon.b', '[v1:172.21.15.53:6789]')] 2024-03-25T19:40:03.361 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.53:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:40:03.362 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:03.362 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:40:03.416 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.18:6789]' --addv c '[v1:172.21.15.18:6790]' --addv b '[v1:172.21.15.53:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:03.501 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:03.501 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: generated fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:03.501 INFO:teuthology.orchestra.run.smithi018.stdout:setting min_mon_release = quincy 2024-03-25T19:40:03.501 INFO:teuthology.orchestra.run.smithi018.stdout:epoch 0 2024-03-25T19:40:03.501 INFO:teuthology.orchestra.run.smithi018.stdout:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:03.502 INFO:teuthology.orchestra.run.smithi018.stdout:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:03.502 INFO:teuthology.orchestra.run.smithi018.stdout:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:03.502 INFO:teuthology.orchestra.run.smithi018.stdout:min_mon_release 17 (quincy) 2024-03-25T19:40:03.502 INFO:teuthology.orchestra.run.smithi018.stdout:election_strategy: 3 2024-03-25T19:40:03.502 INFO:teuthology.orchestra.run.smithi018.stdout:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:03.502 INFO:teuthology.orchestra.run.smithi018.stdout:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:03.502 INFO:teuthology.orchestra.run.smithi018.stdout:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:03.502 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:40:03.503 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:40:03.517 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e9758276-eabb-4964-9819-31fbe65d0466... 2024-03-25T19:40:03.518 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:40:03.560 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:40:03.607 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2024-03-25T19:40:03.607 INFO:teuthology.orchestra.run.smithi018.stdout: chdir = "" 2024-03-25T19:40:03.607 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:40:03.607 INFO:teuthology.orchestra.run.smithi018.stdout: auth supported = cephx 2024-03-25T19:40:03.607 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.607 INFO:teuthology.orchestra.run.smithi018.stdout: filestore xattr use omap = true 2024-03-25T19:40:03.607 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.607 INFO:teuthology.orchestra.run.smithi018.stdout: mon clock drift allowed = 1.000 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: osd crush chooseleaf type = 0 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: auth debug = true 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on old message = true 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on bug = true 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: mon pg warn max object skew = 0 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default size = 2 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow primary affinity = true 2024-03-25T19:40:03.608 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow pg remap = true 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on no sortbitwise = false 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on too few osds = false 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon_allow_pool_size_one = true 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: osd default data pool replay window = 5 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: mon allow pool delete = true 2024-03-25T19:40:03.609 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: mon cluster log file level = debug 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: debug asserts on shutdown = true 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: mon health detail to clog = false 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: mon host = "[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.53:6789]" 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: mon client directed command retry = 5 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: mon election default strategy = 3 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: ms bind msgr2 = False 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: ms inject socket failures = 5000 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: ms type = async 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: osd heartbeat grace = 80 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: fsid = e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout:[osd] 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: osd journal size = 100 2024-03-25T19:40:03.610 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub load threshold = 5.0 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub max interval = 600 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd recover clone overlap = true 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd recovery max chunk = 1048576 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug shutdown = true 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug op order = true 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify stray on activate = true 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug trim objects = true 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd open classes on start = true 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug pg log writeout = true 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:40:03.611 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: osd map max advance = 10 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: journal zero on create = true 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: filestore apply finisher threads = 3 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: bdev debug aio = true 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug misdirected ops = true 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: bdev async discard = True 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: bdev enable discard = True 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore allocator = stupid 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore block size = 96636764160 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore fsck on mount = True 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore zero block detection = True 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluefs = 1 2024-03-25T19:40:03.612 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluestore = 1 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: debug monc = 20 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: debug osd = 20 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: debug rocksdb = 10 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd full ratio = 0.9 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd class default list = * 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd class load list = * 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd client watch timeout = 120 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify cached snaps = True 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify missing on start = True 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd fast shutdown = False 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd max markdown count = 1000 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd mclock override recovery settings = True 2024-03-25T19:40:03.613 INFO:teuthology.orchestra.run.smithi018.stdout: osd objectstore = bluestore 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: osd op queue = debug_random 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: osd op queue cut off = debug_random 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: osd shutdown pgref assert = True 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr] 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: debug mgr = 20 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: mgr/telemetry/nag = false 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout:[mon] 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-25T19:40:03.614 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: debug paxos = 20 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: mon data avail warn = 5 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: mon mgr mkfs grace = 240 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd reporter subtree level = osd 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd prime pg temp = true 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:40:03.615 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: # 1m isn't quite enough 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon min osdmap epochs = 50 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd crush smoke test = False 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune interval = 2 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune min = 15 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon scrub interval = 300 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on pool no app = False 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: paxos service trim min = 10 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout:[client] 2024-03-25T19:40:03.616 INFO:teuthology.orchestra.run.smithi018.stdout: rgw cache enabled = true 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable ops log = true 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable usage log = true 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout: debug monc = 20 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout: debug objecter = 20 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout: debug rados = 20 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.a] 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.c] 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.b] 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-03-25T19:40:03.617 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:40:03.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:40:03.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.53:6789]" 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: mon client directed command retry = 5 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: mon election default strategy = 3 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: ms bind msgr2 = False 2024-03-25T19:40:03.620 INFO:teuthology.orchestra.run.smithi053.stdout: ms inject socket failures = 5000 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: ms type = async 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd heartbeat grace = 80 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-03-25T19:40:03.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug trim objects = true 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.622 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: bdev async discard = True 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: bdev enable discard = True 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore allocator = stupid 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore zero block detection = True 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 1 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 1 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug monc = 20 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 10 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:40:03.623 INFO:teuthology.orchestra.run.smithi053.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd class default list = * 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd class load list = * 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd client watch timeout = 120 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify cached snaps = True 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify missing on start = True 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd fast shutdown = False 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd max markdown count = 1000 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock override recovery settings = True 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd op queue = debug_random 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd op queue cut off = debug_random 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: osd shutdown pgref assert = True 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-25T19:40:03.624 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-03-25T19:40:03.625 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: mon min osdmap epochs = 50 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd crush smoke test = False 2024-03-25T19:40:03.626 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune interval = 2 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune min = 15 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: mon scrub interval = 300 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on pool no app = False 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: paxos service trim min = 10 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: debug monc = 20 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: debug objecter = 20 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout: debug rados = 20 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-03-25T19:40:03.627 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2024-03-25T19:40:03.628 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2024-03-25T19:40:03.629 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:40:03.629 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:40:03.683 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:40:03.683 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:03.683 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:40:03.698 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:03.698 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:40:03.753 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:40:03.753 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:03.753 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:40:03.753 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:03.832 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:03.832 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:03.887 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:40:03.887 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:03.887 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:40:03.887 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:03.928 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:03.928 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:03.983 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:40:03.984 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:40:03.984 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:40:04.038 INFO:teuthology.orchestra.run.smithi018.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:40:04.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:40:04.105 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:40:04.108 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:40:04.108 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:40:04.108 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:40:04.150 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:40:04.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:40:04.208 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:40:04.223 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:40:04.223 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:40:04.223 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:04.250 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:04.250 INFO:tasks.ceph:role: osd.0 2024-03-25T19:40:04.250 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:40:04.250 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:40:04.316 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:04.316 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:04.316 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:04.316 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:04.316 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:04.316 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:04.317 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:04.317 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:04.317 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:04.317 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:04.349 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-25T19:40:04.351 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-25T19:40:04.351 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:04.422 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:04.493 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:04.560 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:04.560 INFO:tasks.ceph:role: osd.1 2024-03-25T19:40:04.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:40:04.561 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:40:04.626 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:04.626 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:04.627 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:04.627 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:04.627 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:04.627 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:04.627 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:04.627 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:04.627 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:04.627 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:04.659 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-25T19:40:04.661 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-25T19:40:04.661 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:04.731 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:04.802 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:04.869 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:04.869 INFO:tasks.ceph:role: osd.2 2024-03-25T19:40:04.869 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:40:04.869 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:40:04.935 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:04.935 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:04.935 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:04.935 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:04.935 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:04.935 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:04.935 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:04.936 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:04.936 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:04.936 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:04.968 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-25T19:40:04.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-25T19:40:04.969 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:05.044 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:05.115 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:05.182 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:05.182 INFO:tasks.ceph:role: osd.3 2024-03-25T19:40:05.182 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:40:05.182 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:40:05.247 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:05.248 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:05.280 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-25T19:40:05.282 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-25T19:40:05.282 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:05.352 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:05.422 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:05.525 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:05.525+0000 7f2bdc945640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:40:05.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:05.526+0000 7f2bdc945640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:40:05.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:05.526+0000 7f2bdc945640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:05.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:05.526+0000 7f2bdc945640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:05.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:05.526+0000 7f2bdc945640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:05.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:05.526+0000 7f2bdc945640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:40:08.372 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:08.403 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:08.493 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:08.494+0000 7fb44f6a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:40:08.494 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:08.494+0000 7fb44f6a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:40:08.494 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:08.494+0000 7fb44f6a8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:08.494 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:08.494+0000 7fb44f6a8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:08.494 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:08.494+0000 7fb44f6a8640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:08.494 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:08.494+0000 7fb44f6a8640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:40:11.343 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:11.373 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:11.464 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:11.464+0000 7f0c27ace640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:40:11.464 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:11.464+0000 7f0c27ace640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:40:11.464 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:11.464+0000 7f0c27ace640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:11.464 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:11.464+0000 7f0c27ace640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:11.464 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:11.464+0000 7f0c27ace640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:11.464 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:11.464+0000 7f0c27ace640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:40:14.305 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:14.336 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:14.426 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:14.426+0000 7fc46380d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:40:14.426 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:14.426+0000 7fc46380d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:40:14.427 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:14.426+0000 7fc46380d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:14.427 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:14.427+0000 7fc46380d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:14.427 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:14.427+0000 7fc46380d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:14.427 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:14.427+0000 7fc46380d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:40:17.267 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:17.298 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:17.330 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:17.330 INFO:tasks.ceph:role: osd.4 2024-03-25T19:40:17.331 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:40:17.331 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:40:17.401 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:17.401 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:17.401 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:17.401 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:17.401 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:17.402 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:17.402 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:17.402 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:17.402 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:17.402 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:17.434 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-25T19:40:17.436 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-25T19:40:17.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:17.508 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:17.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:17.648 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:17.648 INFO:tasks.ceph:role: osd.5 2024-03-25T19:40:17.649 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:40:17.649 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:40:17.716 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:17.716 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:17.716 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:17.716 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:17.716 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:17.716 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:17.716 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:17.717 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:17.717 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:17.717 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:17.748 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-25T19:40:17.750 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-25T19:40:17.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:17.831 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:17.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:17.970 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:17.970 INFO:tasks.ceph:role: osd.6 2024-03-25T19:40:17.970 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:40:17.971 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:40:18.036 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:18.036 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:18.036 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:18.036 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:18.036 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:18.037 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:18.037 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:18.037 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:18.037 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:18.037 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:18.069 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-25T19:40:18.071 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-25T19:40:18.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:18.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:18.212 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:18.279 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:18.279 INFO:tasks.ceph:role: osd.7 2024-03-25T19:40:18.279 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:40:18.279 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:40:18.345 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:18.345 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:18.345 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:18.345 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:18.345 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:18.345 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:18.345 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:18.345 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:18.346 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:18.346 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:18.378 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-25T19:40:18.380 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-25T19:40:18.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:18.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:18.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:18.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:18.626+0000 7f6e80814640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:40:18.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:18.627+0000 7f6e80814640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:40:18.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:18.627+0000 7f6e80814640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:18.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:18.627+0000 7f6e80814640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:18.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:18.627+0000 7f6e80814640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:18.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:18.627+0000 7f6e80814640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:40:21.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:21.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:21.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:21.584+0000 7f34d51f8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:40:21.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:21.584+0000 7f34d51f8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:40:21.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:21.584+0000 7f34d51f8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:21.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:21.584+0000 7f34d51f8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:21.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:21.584+0000 7f34d51f8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:21.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:21.585+0000 7f34d51f8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:40:24.435 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:24.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:24.555+0000 7fd22dac2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:40:24.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:24.555+0000 7fd22dac2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:40:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:24.556+0000 7fd22dac2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:24.556+0000 7fd22dac2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:24.556+0000 7fd22dac2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:24.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:24.556+0000 7fd22dac2640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:40:27.398 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:27.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:27.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:27.521+0000 7f4a0dc68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:40:27.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:27.521+0000 7f4a0dc68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:40:27.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:27.521+0000 7f4a0dc68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:27.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:27.521+0000 7f4a0dc68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:27.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:27.521+0000 7f4a0dc68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:27.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:27.521+0000 7f4a0dc68640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:40:30.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:30.392 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:40:30.392 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:30.392 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:40:30.423 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:30.424 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:40:30.493 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:30.493 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:40:30.562 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:30.562 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:40:30.630 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:30.630 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:40:30.699 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:30.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:40:30.730 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:30.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:40:30.800 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:30.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:40:30.870 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:30.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:40:30.939 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:30.939 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:40:31.008 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:31.008 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:40:31.025 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:31.025 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:40:31.063 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:40:31.064 DEBUG:teuthology.orchestra.run.smithi018:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:40:31.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:40:31.106 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr.y] 2024-03-25T19:40:31.106 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAU0wFmL+FgAhAALMiOGKaITaGyx53I8JWHAw== 2024-03-25T19:40:31.106 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.0] 2024-03-25T19:40:31.106 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAV0wFmJGBlHxAAGueouUbDQRptb3Amo5Koog== 2024-03-25T19:40:31.106 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.1] 2024-03-25T19:40:31.106 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAY0wFmSb1+HRAAtvEkL1XS9T6CJUSGJCjBYQ== 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.2] 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAb0wFm/f+3GxAALGfblj5suMYMSVA9dNnZaw== 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.3] 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAe0wFmvM58GRAArYmClgKQ9pwfQfcJmA3TlQ== 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr.x] 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAU0wFmY6hnBhAARRToDwG+3UjimgqjKGI65Q== 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.4] 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAi0wFmlA1qJRAAppEUumq56UHIB0nD2EhIvw== 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.5] 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAl0wFm8UX1IhAAhuE36IILgms/fXauLA5ZBg== 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.6] 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAo0wFmdHM8IRAAgwYE6n8k7Q1PdD85R82xSQ== 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.7] 2024-03-25T19:40:31.107 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAr0wFmHK8gHxAAusU31NVBUE0p3/j68B5J3A== 2024-03-25T19:40:31.108 INFO:teuthology.orchestra.run.smithi018.stdout:[client.0] 2024-03-25T19:40:31.108 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAU0wFmfgoPCRAAr+NxxwfdE+IVIpeU0j0NeA== 2024-03-25T19:40:31.108 INFO:teuthology.orchestra.run.smithi018.stdout:[client.1] 2024-03-25T19:40:31.108 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAU0wFm0Rx/DBAAfbMFux6HbUiKOeF4JEHCJg== 2024-03-25T19:40:31.129 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.y] 2024-03-25T19:40:31.129 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAU0wFmL+FgAhAALMiOGKaITaGyx53I8JWHAw== 2024-03-25T19:40:31.129 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2024-03-25T19:40:31.129 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAV0wFmJGBlHxAAGueouUbDQRptb3Amo5Koog== 2024-03-25T19:40:31.129 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2024-03-25T19:40:31.129 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAY0wFmSb1+HRAAtvEkL1XS9T6CJUSGJCjBYQ== 2024-03-25T19:40:31.129 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAb0wFm/f+3GxAALGfblj5suMYMSVA9dNnZaw== 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.3] 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAe0wFmvM58GRAArYmClgKQ9pwfQfcJmA3TlQ== 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAU0wFmY6hnBhAARRToDwG+3UjimgqjKGI65Q== 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.4] 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAi0wFmlA1qJRAAppEUumq56UHIB0nD2EhIvw== 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.5] 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAl0wFm8UX1IhAAhuE36IILgms/fXauLA5ZBg== 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.6] 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAo0wFmdHM8IRAAgwYE6n8k7Q1PdD85R82xSQ== 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.7] 2024-03-25T19:40:31.130 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAr0wFmHK8gHxAAusU31NVBUE0p3/j68B5J3A== 2024-03-25T19:40:31.131 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-03-25T19:40:31.131 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAU0wFmfgoPCRAAr+NxxwfdE+IVIpeU0j0NeA== 2024-03-25T19:40:31.131 INFO:teuthology.orchestra.run.smithi053.stdout:[client.1] 2024-03-25T19:40:31.131 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAU0wFm0Rx/DBAAfbMFux6HbUiKOeF4JEHCJg== 2024-03-25T19:40:31.132 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:31.152 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:31.225 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.227 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.279 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.332 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.386 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.388 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.439 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:31.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:31.492 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.494 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.545 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.548 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.599 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.601 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.653 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:31.706 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:31.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:31.763 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:31.765 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:31.819 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:40:31.819 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:40:31.845 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:31.941 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:40:31.967 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:40:32.033 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:32.130 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:40:32.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:40:32.183 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:32.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:40:32.306 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:32.309 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:32.360 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:40:32.360 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:32.361 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:40:32.361 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:40:32.363 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:40:32.363 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:32.364 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:40:32.364 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:40:32.365 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:40:32.366 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:32.366 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:40:32.366 DEBUG:teuthology.orchestra.run.smithi053:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:40:32.402 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:40:32.402 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:40:32.403 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:40:32.403 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:40:32.405 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:40:32.405 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:40:32.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:40:32.407 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:40:32.407 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:40:32.407 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.524+0000 7fcc29ba3640 10 monclient: get_monmap_and_config 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.524+0000 7fcc29ba3640 10 monclient: build_initial_monmap 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.524+0000 7fcc29ba3640 10 monclient: monmap: 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:32.525422+0000 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:32.525422+0000 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:32.524 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:32.525 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 1 Processor -- start 2024-03-25T19:40:32.525 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 1 -- start start 2024-03-25T19:40:32.525 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 10 monclient: init 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 10 monclient(hunting): picked mon.noname-c con 0x7fcc240a9920 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 10 monclient(hunting): picked mon.noname-b con 0x7fcc240b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 10 monclient(hunting): picked mon.noname-a con 0x7fcc24146450 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2406b100 con 0x7fcc24146450 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240b2940 con 0x7fcc240a9920 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2413fec0 con 0x7fcc240b45a0 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.525+0000 7fcc29ba3640 10 monclient(hunting): authenticate will time out at 4859.652726s 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.526+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc24145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.526+0000 7fcc23fff640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc2414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:32.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.526+0000 7fcc237fe640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc2413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:32.726 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.727+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc24145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:32.726 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.727+0000 7fcc23fff640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc2414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:32.727 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:32.727+0000 7fcc237fe640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc2413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:33.127 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:33.128+0000 7fcc23fff640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc2414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:33.127 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:33.128+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc24145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:33.128 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:33.128+0000 7fcc237fe640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc2413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:33.928 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:33.929+0000 7fcc23fff640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc2414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:33.928 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:33.929+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc24145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:33.928 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:33.929+0000 7fcc237fe640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc2413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:35.525 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.525+0000 7fcc217fa640 10 monclient(hunting): tick 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc24145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc2413f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc2414dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcc24146450 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fcc240b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-c con 0x7fcc240a9920 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2406b100 con 0x7fcc24146450 2024-03-25T19:40:35.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240b2940 con 0x7fcc240a9920 2024-03-25T19:40:35.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2413fec0 con 0x7fcc240b45a0 2024-03-25T19:40:35.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc237fe640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc100056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:35.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc217fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:35.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:35.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.526+0000 7fcc23fff640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:35.726 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.727+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:35.727 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.727+0000 7fcc237fe640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc100056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:35.727 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:35.727+0000 7fcc23fff640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:36.127 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:36.128+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:36.127 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:36.128+0000 7fcc237fe640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc100056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:36.127 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:36.128+0000 7fcc23fff640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:36.928 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:36.929+0000 7fcc237fe640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc100056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:36.929 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:36.929+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:36.929 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:36.929+0000 7fcc23fff640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 10 monclient(hunting): tick 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc100056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-c con 0x7fcc240a9920 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:38.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcc24146450 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:38.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fcc240b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:38.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc10010e20 con 0x7fcc24146450 2024-03-25T19:40:38.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2406b100 con 0x7fcc240a9920 2024-03-25T19:40:38.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240b2940 con 0x7fcc240b45a0 2024-03-25T19:40:38.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc217fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:38.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:38.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc23fff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:38.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.526+0000 7fcc237fe640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:38.726 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.727+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:38.726 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.727+0000 7fcc23fff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:38.727 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:38.727+0000 7fcc237fe640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:39.127 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:39.128+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:39.127 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:39.128+0000 7fcc23fff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:39.127 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:39.128+0000 7fcc237fe640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:39.829 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:00:07.360 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.829 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:00:07.360 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.861 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:00:07.357 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:39.928 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:39.929+0000 7fcc22ffd640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T19:40:39.928 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:39.929+0000 7fcc23fff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T19:40:39.928 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:39.929+0000 7fcc237fe640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-25T19:40:40.156 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:00:07.687 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:40.156 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:00:07.687 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:40.187 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:00:07.684 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 10 monclient(hunting): tick 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-c con 0x7fcc240a9920 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcc24146450 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:41.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fcc240b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:41.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2413fec0 con 0x7fcc24146450 2024-03-25T19:40:41.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc10010e20 con 0x7fcc240a9920 2024-03-25T19:40:41.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2406b100 con 0x7fcc240b45a0 2024-03-25T19:40:41.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.526+0000 7fcc217fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:41.639 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.639+0000 7fcc23fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 0x7fcc10008ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:47154/0 (socket says 172.21.15.18:47154) 2024-03-25T19:40:41.639 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.639+0000 7fcc23fff640 1 -- 172.21.15.18:0/1564646629 learned_addr learned my addr 172.21.15.18:0/1564646629 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:40:41.836 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.836+0000 7fcc237fe640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T19:40:41.836 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.837+0000 7fcc237fe640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:40:41.836 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.837+0000 7fcc237fe640 1 --1- 172.21.15.18:0/1564646629 >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 0x7fcc1000fdc0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:40:41.864 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.865+0000 7fcc23fff640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:40:41.864 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.865+0000 7fcc23fff640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:40:41.864 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.865+0000 7fcc23fff640 1 --1- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 0x7fcc10008ea0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-25T19:40:41.864 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.865+0000 7fcc23fff640 1 --1- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 0x7fcc10008ea0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:40:41.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.865+0000 7fcc227fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.18:6790/0 2024-03-25T19:40:41.902 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.902+0000 7fcc22ffd640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T19:40:41.902 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.902+0000 7fcc22ffd640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:40:41.902 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.902+0000 7fcc22ffd640 1 --1- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 0x7fcc10005630 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T19:40:41.902 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.902+0000 7fcc22ffd640 1 --1- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 0x7fcc10005630 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:40:41.902 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:41.902+0000 7fcc227fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.18:6789/0 2024-03-25T19:40:44.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 10 monclient(hunting): tick 2024-03-25T19:40:44.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:44.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:44.526 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.53:6789/0 conn(0x7fcc240a9920 legacy=0x7fcc1000fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fcc240b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-c con 0x7fcc1000c6e0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcc24146450 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240b2940 con 0x7fcc24146450 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1000cbb0 con 0x7fcc1000c6e0 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1000ee10 con 0x7fcc240b45a0 2024-03-25T19:40:44.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:44.526+0000 7fcc217fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.526+0000 7fcc217fa640 10 monclient(hunting): tick 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.526+0000 7fcc217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.526+0000 7fcc217fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.526+0000 7fcc217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10010590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.53:6789/0 conn(0x7fcc1000c6e0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.526+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6789/0 conn(0x7fcc24146450 legacy=0x7fcc10005630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.527+0000 7fcc217fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.527+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcc1000efa0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.527+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-c con 0x7fcc1000c6e0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:47.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.527+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fcc240b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:47.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc10010070 con 0x7fcc1000efa0 2024-03-25T19:40:47.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1000faf0 con 0x7fcc1000c6e0 2024-03-25T19:40:47.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1000fd20 con 0x7fcc240b45a0 2024-03-25T19:40:47.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:47.527+0000 7fcc217fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 10 monclient(hunting): tick 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10010590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.53:6789/0 conn(0x7fcc1000c6e0 legacy=0x7fcc10008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6789/0 conn(0x7fcc1000efa0 legacy=0x7fcc1000f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:50.527 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fcc10009670 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:50.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fcc240b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:50.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 10 monclient(hunting): picked mon.noname-c con 0x7fcc1000c6e0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:50.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc10004630 con 0x7fcc10009670 2024-03-25T19:40:50.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1000a300 con 0x7fcc1000c6e0 2024-03-25T19:40:50.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100057d0 con 0x7fcc240b45a0 2024-03-25T19:40:50.528 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.527+0000 7fcc217fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:50.540 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.541+0000 7fcc22ffd640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10010570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T19:40:50.540 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.541+0000 7fcc22ffd640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10010570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:40:50.540 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.541+0000 7fcc22ffd640 1 --1- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 0x7fcc10010570 :-1 s=OPENED pgs=18 cs=1 l=1).handle_message read tag failed 2024-03-25T19:40:50.541 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.541+0000 7fcc22ffd640 1 --1- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 0x7fcc10010570 :-1 s=OPENED pgs=18 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:40:50.541 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.541+0000 7fcc227fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.18:6790/0 2024-03-25T19:40:50.717 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.717+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472449625 0 0) 0x7fcc10004630 con 0x7fcc10009670 2024-03-25T19:40:50.717 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.717+0000 7fcc227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.717 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.717+0000 7fcc227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.717 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.717+0000 7fcc227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.717 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.717+0000 7fcc227fc640 10 monclient(hunting): my global_id is 4104 2024-03-25T19:40:50.717 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.717+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c003520 con 0x7fcc10009670 2024-03-25T19:40:50.737 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.737+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107100515 0 0) 0x7fcc1000a300 con 0x7fcc1000c6e0 2024-03-25T19:40:50.737 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.737+0000 7fcc227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.737 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.737+0000 7fcc227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.737 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.737+0000 7fcc227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.737 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.737+0000 7fcc227fc640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:40:50.737 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.737+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc10004630 con 0x7fcc1000c6e0 2024-03-25T19:40:50.814 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563800414 0 0) 0x7fcc0c003520 con 0x7fcc10009670 2024-03-25T19:40:50.814 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc10010570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:40:50.814 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.53:6789/0 conn(0x7fcc1000c6e0 legacy=0x7fcc1000f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.814 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:50.814 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 1 monclient: found mon.noname-a 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10005960 con 0x7fcc10009670 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fcc14003f80 con 0x7fcc10009670 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc227fc640 10 monclient: dump: 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:50.815 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.816 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:50.816 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:50.816 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:50.816 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:50.816 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:50.816 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc29ba3640 5 monclient: authenticate success, global_id 4104 2024-03-25T19:40:50.816 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.814+0000 7fcc29ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:50.859 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc140040e0 con 0x7fcc10009670 2024-03-25T19:40:50.859 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:50.859 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc227fc640 1 -- 172.21.15.18:0/1564646629 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fcc14004530 con 0x7fcc10009670 2024-03-25T19:40:50.859 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc29ba3640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:50.859 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc29ba3640 4 set_mon_vals no callback set 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc227fc640 10 monclient: dump: 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:50.860 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc29ba3640 10 monclient: shutdown 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc29ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc29ba3640 1 -- 172.21.15.18:0/1564646629 >> v1:172.21.15.18:6789/0 conn(0x7fcc10009670 legacy=0x7fcc10009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc29ba3640 1 -- 172.21.15.18:0/1564646629 shutdown_connections 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.860+0000 7fcc29ba3640 1 -- 172.21.15.18:0/1564646629 >> 172.21.15.18:0/1564646629 conn(0x7fcc240b7f40 msgr2=0x7fcc240b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.861+0000 7fcc29ba3640 1 -- 172.21.15.18:0/1564646629 shutdown_connections 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.861+0000 7fcc29ba3640 1 -- 172.21.15.18:0/1564646629 wait complete. 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.861+0000 7fcc29ba3640 10 monclient: build_initial_monmap 2024-03-25T19:40:50.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.861+0000 7fcc29ba3640 10 monclient: monmap: 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 0.000000 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:created 0.000000 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.862+0000 7fcc29ba3640 1 librados: starting msgr at 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.862+0000 7fcc29ba3640 1 librados: starting objecter 2024-03-25T19:40:50.863 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.862+0000 7fcc29ba3640 1 Processor -- start 2024-03-25T19:40:50.864 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.862+0000 7fcc29ba3640 1 -- start start 2024-03-25T19:40:50.864 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.862+0000 7fcc29ba3640 1 librados: setting wanted keys 2024-03-25T19:40:50.864 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.862+0000 7fcc29ba3640 1 librados: calling monclient init 2024-03-25T19:40:50.864 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.862+0000 7fcc29ba3640 10 monclient: init 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 10 monclient(hunting): picked mon.noname-a con 0x7fcc10009670 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1000c6e0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 10 monclient(hunting): picked mon.noname-c con 0x7fcc240b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240acfb0 con 0x7fcc10009670 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc241daf80 con 0x7fcc1000c6e0 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc241db110 con 0x7fcc240b45a0 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:50.865 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.863+0000 7fcc29ba3640 10 monclient(hunting): authenticate will time out at 4877.990492s 2024-03-25T19:40:50.877 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.877+0000 7fcc237fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fcc10009670 0x7fcc241da870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:44128/0 (socket says 172.21.15.18:44128) 2024-03-25T19:40:50.877 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.877+0000 7fcc237fe640 1 -- 172.21.15.18:0/2013653250 learned_addr learned my addr 172.21.15.18:0/2013653250 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:40:50.877 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.878+0000 7fcc0bfff640 10 client.?.objecter ms_handle_connect 0x7fcc10009670 2024-03-25T19:40:50.878 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.878+0000 7fcc0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:50.882 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.883+0000 7fcc0bfff640 10 client.?.objecter ms_handle_connect 0x7fcc1000c6e0 2024-03-25T19:40:50.882 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.883+0000 7fcc0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:50.883 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.884+0000 7fcc0bfff640 10 client.?.objecter ms_handle_connect 0x7fcc240b45a0 2024-03-25T19:40:50.883 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.884+0000 7fcc0bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:50.886 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.887+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920478127 0 0) 0x7fcc240acfb0 con 0x7fcc10009670 2024-03-25T19:40:50.886 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.887+0000 7fcc0bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:50.886 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.887+0000 7fcc0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:50.886 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.887+0000 7fcc0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:50.887 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.887+0000 7fcc0bfff640 10 monclient(hunting): my global_id is 4110 2024-03-25T19:40:50.887 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.887+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1000a300 con 0x7fcc10009670 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.954+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2524908417 0 0) 0x7fcc1000a300 con 0x7fcc10009670 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.954+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc240acfb0 con 0x7fcc10009670 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.954+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fcc140044f0 con 0x7fcc10009670 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.954+0000 7fcc0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.954+0000 7fcc0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.954+0000 7fcc0bfff640 10 monclient(hunting): dump: 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:50.954 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:50.955 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:50.955 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:50.955 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:50.955 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:50.955 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.954+0000 7fcc0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1749543220 0 0) 0x7fcc240acfb0 con 0x7fcc10009670 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 >> v1:172.21.15.18:6790/0 conn(0x7fcc240b45a0 legacy=0x7fcc241e1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 >> v1:172.21.15.53:6789/0 conn(0x7fcc1000c6e0 legacy=0x7fcc24141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 1 monclient: found mon.a 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc240ab910 con 0x7fcc10009670 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 10 monclient: _finish_auth 0 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:50.965 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 5 monclient: authenticate success, global_id 4110 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 10 monclient: _renew_subs 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc240abbc0 con 0x7fcc10009670 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 10 monclient: _renew_subs 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc241443d0 con 0x7fcc10009670 2024-03-25T19:40:50.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.965+0000 7fcc29ba3640 1 librados: init done 2024-03-25T19:40:50.967 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.966+0000 7fcc29ba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:50.967 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.966+0000 7fcc29ba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:50.967 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.966+0000 7fcc29ba3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:50.967 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:50.966+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc24144740 con 0x7fcc10009670 2024-03-25T19:40:51.181 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc14004900 con 0x7fcc10009670 2024-03-25T19:40:51.181 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:51.181 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fcc14004b70 con 0x7fcc10009670 2024-03-25T19:40:51.181 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.181 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 10 monclient: dump: 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcc14005120 con 0x7fcc10009670 2024-03-25T19:40:51.182 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc21ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:51.183 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1169930271 0 0) 0x7fcc14006790 con 0x7fcc10009670 2024-03-25T19:40:51.183 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 10 client.4110.objecter ms_dispatch 0x7fcc24000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:40:51.183 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:40:51.183 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:40:51.183 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T19:40:51.183 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcc14006950 con 0x7fcc10009670 2024-03-25T19:40:51.183 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:51.183 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.181+0000 7fcc0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:51.251 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.251+0000 7fcc29ba3640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:51.251 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.251+0000 7fcc29ba3640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:51.251 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.251+0000 7fcc29ba3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:51.251 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.251+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcc2404f730 con 0x7fcc10009670 2024-03-25T19:40:51.399 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:51.400+0000 7fc741237040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:51.407 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.408+0000 7fcc0bfff640 1 -- 172.21.15.18:0/2013653250 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fcc140054b0 con 0x7fcc10009670 2024-03-25T19:40:51.407 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.408+0000 7fcc0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:51.407 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.408+0000 7fcc0bfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:40:51.407 INFO:teuthology.orchestra.run.smithi018.stderr:adjusted tunables profile to default 2024-03-25T19:40:51.408 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.408+0000 7fcc29ba3640 10 librados: watch_flush enter 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.408+0000 7fcc29ba3640 10 librados: watch_flush exit 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.408+0000 7fcc29ba3640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.408+0000 7fcc29ba3640 10 client.4110.objecter successfully canceled tick 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.409+0000 7fcc29ba3640 10 monclient: shutdown 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.409+0000 7fcc29ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.409+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 >> v1:172.21.15.18:6789/0 conn(0x7fcc10009670 legacy=0x7fcc241da870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.409+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 shutdown_connections 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.409+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 >> 172.21.15.18:0/2013653250 conn(0x7fcc240b7f40 msgr2=0x7fcc240b6d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.409+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 shutdown_connections 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.409+0000 7fcc29ba3640 1 -- 172.21.15.18:0/2013653250 wait complete. 2024-03-25T19:40:51.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:51.409+0000 7fcc29ba3640 1 librados: shutdown 2024-03-25T19:40:51.420 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:40:51.420 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:40:51.420 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:51.420 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:40:51.449 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:51.449 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:40:51.469 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:51.470+0000 7fc741237040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:51.495 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:51.495 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:40:51.562 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T19:40:51.562 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:40:51.630 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:51.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:40:51.658 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:51.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:40:51.679 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:51.679+0000 7f123ec3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:51.705 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:51.705 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:40:51.736 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:51.737+0000 7f123ec3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:51.764 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-25T19:40:51.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:40:51.831 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6 0 2024-03-25T19:40:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.972+0000 7fc119132640 10 monclient: get_monmap_and_config 2024-03-25T19:40:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.972+0000 7fc119132640 10 monclient: build_initial_monmap 2024-03-25T19:40:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.972+0000 7fc119132640 10 monclient: monmap: 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:51.973035+0000 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:51.973035+0000 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:51.972 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:51.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.972+0000 7fc119132640 1 Processor -- start 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.972+0000 7fc119132640 1 -- start start 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.972+0000 7fc119132640 10 monclient: init 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 10 monclient(hunting): picked mon.noname-b con 0x7fc1140b4700 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 10 monclient(hunting): picked mon.noname-a con 0x7fc114142a20 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 10 monclient(hunting): picked mon.noname-c con 0x7fc1140a96b0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1140a9d60 con 0x7fc114142a20 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11413ff10 con 0x7fc1140a96b0 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1140aa1f0 con 0x7fc1140b4700 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc119132640 10 monclient(hunting): authenticate will time out at 4886.126105s 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc11359e640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc1140a96b0 0x7fc11413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42590/0 (socket says 172.21.15.53:42590) 2024-03-25T19:40:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.973+0000 7fc11359e640 1 -- 172.21.15.53:0/291927539 learned_addr learned my addr 172.21.15.53:0/291927539 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601141944 0 0) 0x7fc11413ff10 con 0x7fc1140a96b0 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0fc003520 con 0x7fc1140a96b0 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510865544 0 0) 0x7fc1140aa1f0 con 0x7fc1140b4700 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc11413ff10 con 0x7fc1140b4700 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659989830 0 0) 0x7fc1140a9d60 con 0x7fc114142a20 2024-03-25T19:40:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:40:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.976+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1140aa1f0 con 0x7fc114142a20 2024-03-25T19:40:51.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 24412865 0 0) 0x7fc1140aa1f0 con 0x7fc114142a20 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 >> v1:172.21.15.18:6790/0 conn(0x7fc1140b4700 legacy=0x7fc114142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 >> v1:172.21.15.53:6789/0 conn(0x7fc1140a96b0 legacy=0x7fc11413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 1 monclient: found mon.noname-a 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1140b4ae0 con 0x7fc114142a20 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 10 monclient: _finish_auth 0 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc119132640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc1040015b0 con 0x7fc114142a20 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc111d9b640 10 monclient: dump: 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:51.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.980+0000 7fc119132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:51.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc104001710 con 0x7fc114142a20 2024-03-25T19:40:51.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc111d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:51.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc111d9b640 1 -- 172.21.15.53:0/291927539 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc104001b60 con 0x7fc114142a20 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc111d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc111d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc111d9b640 10 monclient: dump: 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc119132640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc119132640 4 set_mon_vals no callback set 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.982+0000 7fc119132640 10 monclient: shutdown 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.983+0000 7fc119132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.983+0000 7fc119132640 1 -- 172.21.15.53:0/291927539 >> v1:172.21.15.18:6789/0 conn(0x7fc114142a20 legacy=0x7fc11413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.983+0000 7fc119132640 1 -- 172.21.15.53:0/291927539 shutdown_connections 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.983+0000 7fc119132640 1 -- 172.21.15.53:0/291927539 >> 172.21.15.53:0/291927539 conn(0x7fc1140668e0 msgr2=0x7fc114066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.983+0000 7fc119132640 1 -- 172.21.15.53:0/291927539 shutdown_connections 2024-03-25T19:40:51.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.983+0000 7fc119132640 1 -- 172.21.15.53:0/291927539 wait complete. 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.983+0000 7fc119132640 10 monclient: build_initial_monmap 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.983+0000 7fc119132640 10 monclient: monmap: 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:51.986 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 1 librados: starting msgr at 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 1 librados: starting objecter 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 1 Processor -- start 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 1 -- start start 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 1 librados: setting wanted keys 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 1 librados: calling monclient init 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 10 monclient: init 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 10 monclient(hunting): picked mon.noname-a con 0x7fc1140a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 10 monclient(hunting): picked mon.noname-c con 0x7fc1140b4700 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 10 monclient(hunting): picked mon.noname-b con 0x7fc114142a20 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.984+0000 7fc119132640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1140adea0 con 0x7fc1140a96b0 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.985+0000 7fc119132640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1140ae180 con 0x7fc114142a20 2024-03-25T19:40:51.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.985+0000 7fc119132640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1140ae460 con 0x7fc1140b4700 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.985+0000 7fc119132640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.985+0000 7fc119132640 10 monclient(hunting): authenticate will time out at 4886.137790s 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.985+0000 7fc112d9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc1140a96b0 0x7fc11413aa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.53:48834/0 (socket says 172.21.15.53:48834) 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.985+0000 7fc112d9d640 1 -- 172.21.15.53:0/876372632 learned_addr learned my addr 172.21.15.53:0/876372632 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.985+0000 7fc0fb7fe640 10 client.?.objecter ms_handle_connect 0x7fc1140a96b0 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.985+0000 7fc0fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.988+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340813514 0 0) 0x7fc1140adea0 con 0x7fc1140a96b0 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.988+0000 7fc0fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.988+0000 7fc0fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:51.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.988+0000 7fc0fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:51.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.988+0000 7fc0fb7fe640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:40:51.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.988+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc100004eb0 con 0x7fc1140a96b0 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.993+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 940240241 0 0) 0x7fc100004eb0 con 0x7fc1140a96b0 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.993+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1140adea0 con 0x7fc1140a96b0 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.993+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc108001a10 con 0x7fc1140a96b0 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.993+0000 7fc0fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.993+0000 7fc0fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.993+0000 7fc0fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:51.993 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:51.994 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:51.994 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:51.994 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:51.994 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:51.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.993+0000 7fc0fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2813729219 0 0) 0x7fc1140adea0 con 0x7fc1140a96b0 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 >> v1:172.21.15.18:6790/0 conn(0x7fc1140b4700 legacy=0x7fc114134c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 >> v1:172.21.15.53:6789/0 conn(0x7fc114142a20 legacy=0x7fc114135330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 1 monclient: found mon.a 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc114135a40 con 0x7fc1140a96b0 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc0fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:51.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:40:51.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 10 monclient: _renew_subs 2024-03-25T19:40:51.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:51.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1140ad020 con 0x7fc1140a96b0 2024-03-25T19:40:51.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:51.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 10 monclient: _renew_subs 2024-03-25T19:40:51.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:51.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11413b350 con 0x7fc1140a96b0 2024-03-25T19:40:51.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.996+0000 7fc119132640 1 librados: init done 2024-03-25T19:40:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.997+0000 7fc119132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.997+0000 7fc119132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.997+0000 7fc119132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:51.997+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11404f730 con 0x7fc1140a96b0 2024-03-25T19:40:52.077 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.078+0000 7fc741237040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:52.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc108001d10 con 0x7fc1140a96b0 2024-03-25T19:40:52.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:52.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc108003a60 con 0x7fc1140a96b0 2024-03-25T19:40:52.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 10 monclient: dump: 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc108004010 con 0x7fc1140a96b0 2024-03-25T19:40:52.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc110d99640 4 set_mon_vals no callback set 2024-03-25T19:40:52.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 606076809 0 0) 0x7fc108006650 con 0x7fc1140a96b0 2024-03-25T19:40:52.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 10 client.4119.objecter ms_dispatch 0x7fc114000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:40:52.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:40:52.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:40:52.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:40:52.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc1080069c0 con 0x7fc1140a96b0 2024-03-25T19:40:52.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:52.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.102+0000 7fc0fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:52.133 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.134+0000 7fc741237040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:52.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.177+0000 7fc119132640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6", "id": 0}] 2024-03-25T19:40:52.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.177+0000 7fc119132640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6", "id": 0}] 2024-03-25T19:40:52.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.177+0000 7fc119132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.177+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6", "id": 0} v 0) v1 -- 0x7fc1140b1d30 con 0x7fc1140a96b0 2024-03-25T19:40:52.180 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.180+0000 7fc741237040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:52.235 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.236+0000 7fc741237040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:52.282 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.282+0000 7fc741237040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:52.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.286+0000 7fc0fb7fe640 1 -- 172.21.15.53:0/876372632 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3348141699 0 732707515) 0x7fc1080057c0 con 0x7fc1140a96b0 2024-03-25T19:40:52.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.286+0000 7fc0fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6", "id": 0}] 2024-03-25T19:40:52.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.286+0000 7fc0fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:52.286 INFO:teuthology.orchestra.run.smithi053.stdout:0 2024-03-25T19:40:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 10 librados: watch_flush enter 2024-03-25T19:40:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 10 librados: watch_flush exit 2024-03-25T19:40:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:40:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:40:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 10 monclient: shutdown 2024-03-25T19:40:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 >> v1:172.21.15.18:6789/0 conn(0x7fc1140a96b0 legacy=0x7fc11413aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 shutdown_connections 2024-03-25T19:40:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 >> 172.21.15.53:0/876372632 conn(0x7fc1140668e0 msgr2=0x7fc11413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 shutdown_connections 2024-03-25T19:40:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 1 -- 172.21.15.53:0/876372632 wait complete. 2024-03-25T19:40:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.287+0000 7fc119132640 1 librados: shutdown 2024-03-25T19:40:52.298 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 8dacde5d-1a0c-4aca-a176-862be9c3a92f 1 2024-03-25T19:40:52.312 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.313+0000 7f123ec3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:52.371 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.371+0000 7f123ec3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:52.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.398+0000 7f6a10552640 10 monclient: get_monmap_and_config 2024-03-25T19:40:52.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.398+0000 7f6a10552640 10 monclient: build_initial_monmap 2024-03-25T19:40:52.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.398+0000 7f6a10552640 10 monclient: monmap: 2024-03-25T19:40:52.398 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:52.398 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.398 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:52.399742+0000 2024-03-25T19:40:52.399 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:52.399742+0000 2024-03-25T19:40:52.399 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:52.399 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.399 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:52.399 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:52.399 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:52.399 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.399+0000 7f6a10552640 1 Processor -- start 2024-03-25T19:40:52.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.399+0000 7f6a10552640 1 -- start start 2024-03-25T19:40:52.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.399+0000 7f6a10552640 10 monclient: init 2024-03-25T19:40:52.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:52.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 10 monclient(hunting): picked mon.noname-c con 0x7f6a080b4700 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 10 monclient(hunting): picked mon.noname-a con 0x7f6a08142a20 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 10 monclient(hunting): picked mon.noname-b con 0x7f6a080a96b0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a080a9d60 con 0x7f6a08142a20 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0813ff10 con 0x7f6a080b4700 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a080aa1f0 con 0x7f6a080a96b0 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a10552640 10 monclient(hunting): authenticate will time out at 4886.553007s 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a0e2ee640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6a080b4700 0x7f6a08142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42602/0 (socket says 172.21.15.53:42602) 2024-03-25T19:40:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.400+0000 7f6a0e2ee640 1 -- 172.21.15.53:0/69748519 learned_addr learned my addr 172.21.15.53:0/69748519 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:52.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619242577 0 0) 0x7f6a0813ff10 con 0x7f6a080b4700 2024-03-25T19:40:52.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:40:52.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f4003520 con 0x7f6a080b4700 2024-03-25T19:40:52.403 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.404+0000 7fc741237040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:52.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877824293 0 0) 0x7f6a080a9d60 con 0x7f6a08142a20 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0813ff10 con 0x7f6a08142a20 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711006918 0 0) 0x7f6a080aa1f0 con 0x7f6a080a96b0 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:40:52.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.403+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a080a9d60 con 0x7f6a080a96b0 2024-03-25T19:40:52.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 822574921 0 0) 0x7f6a0813ff10 con 0x7f6a08142a20 2024-03-25T19:40:52.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 >> v1:172.21.15.18:6790/0 conn(0x7f6a080a96b0 legacy=0x7f6a0813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 >> v1:172.21.15.53:6789/0 conn(0x7f6a080b4700 legacy=0x7f6a08142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 10 monclient: _finish_hunting 0 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 1 monclient: found mon.noname-a 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a080b4ae0 con 0x7f6a08142a20 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 10 monclient: _finish_auth 0 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 10 monclient: _check_auth_tickets 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f69f8001670 con 0x7f6a08142a20 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a0d2ec640 10 monclient: dump: 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.407 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.408 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a10552640 5 monclient: authenticate success, global_id 4122 2024-03-25T19:40:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.406+0000 7f6a10552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:52.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.408+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69f80017d0 con 0x7f6a08142a20 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.408+0000 7f6a0d2ec640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.408+0000 7f6a0d2ec640 1 -- 172.21.15.53:0/69748519 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f69f8001be0 con 0x7f6a08142a20 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.408+0000 7f6a0d2ec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.408+0000 7f6a0d2ec640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.408+0000 7f6a0d2ec640 10 monclient: dump: 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.408+0000 7f6a10552640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.408+0000 7f6a10552640 4 set_mon_vals no callback set 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.409+0000 7f6a10552640 10 monclient: shutdown 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.409+0000 7f6a10552640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.409+0000 7f6a10552640 1 -- 172.21.15.53:0/69748519 >> v1:172.21.15.18:6789/0 conn(0x7f6a08142a20 legacy=0x7f6a0813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.409+0000 7f6a10552640 1 -- 172.21.15.53:0/69748519 shutdown_connections 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.409+0000 7f6a10552640 1 -- 172.21.15.53:0/69748519 >> 172.21.15.53:0/69748519 conn(0x7f6a080668e0 msgr2=0x7f6a08066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.409+0000 7f6a10552640 1 -- 172.21.15.53:0/69748519 shutdown_connections 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.409+0000 7f6a10552640 1 -- 172.21.15.53:0/69748519 wait complete. 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 10 monclient: build_initial_monmap 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 10 monclient: monmap: 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-03-25T19:40:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 1 librados: starting msgr at 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 1 librados: starting objecter 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 1 Processor -- start 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 1 -- start start 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 1 librados: setting wanted keys 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 1 librados: calling monclient init 2024-03-25T19:40:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 10 monclient: init 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 10 monclient(hunting): picked mon.noname-b con 0x7f6a080a96b0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.410+0000 7f6a10552640 10 monclient(hunting): picked mon.noname-a con 0x7f6a080b4700 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f6a10552640 10 monclient(hunting): picked mon.noname-c con 0x7f6a08142a20 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f6a10552640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0813c9b0 con 0x7f6a080b4700 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f6a10552640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0813cc90 con 0x7f6a080a96b0 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f6a10552640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a081d78f0 con 0x7f6a08142a20 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f6a10552640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f6a10552640 10 monclient(hunting): authenticate will time out at 4886.563800s 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f6a0e2ee640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6a080a96b0 0x7f6a081d71e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42614/0 (socket says 172.21.15.53:42614) 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f6a0e2ee640 1 -- 172.21.15.53:0/3469406239 learned_addr learned my addr 172.21.15.53:0/3469406239 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f69feffd640 10 client.?.objecter ms_handle_connect 0x7f6a080a96b0 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f69feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f69feffd640 10 client.?.objecter ms_handle_connect 0x7f6a080b4700 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.411+0000 7f69feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 client.?.objecter ms_handle_connect 0x7f6a08142a20 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023120714 0 0) 0x7f6a0813cc90 con 0x7f6a080a96b0 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec004eb0 con 0x7f6a080a96b0 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132439383 0 0) 0x7f6a0813c9b0 con 0x7f6a080b4700 2024-03-25T19:40:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:40:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.413+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0813cc90 con 0x7f6a080b4700 2024-03-25T19:40:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.415+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516843336 0 0) 0x7f6a081d78f0 con 0x7f6a08142a20 2024-03-25T19:40:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.415+0000 7f69feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.415+0000 7f69feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.415+0000 7f69feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.415+0000 7f69feffd640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:40:52.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.415+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0813c9b0 con 0x7f6a08142a20 2024-03-25T19:40:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.419+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120265505 0 0) 0x7f6a0813cc90 con 0x7f6a080b4700 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.419+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a081d78f0 con 0x7f6a080b4700 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.419+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f69f8001ae0 con 0x7f6a080b4700 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.419+0000 7f69feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.419+0000 7f69feffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.419+0000 7f69feffd640 10 monclient(hunting): dump: 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.420 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.419+0000 7f69feffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:52.420 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.420+0000 7f123ec3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3469715339 0 0) 0x7f6a081d78f0 con 0x7f6a080b4700 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 >> v1:172.21.15.18:6790/0 conn(0x7f6a08142a20 legacy=0x7f6a0813c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 >> v1:172.21.15.53:6789/0 conn(0x7f6a080a96b0 legacy=0x7f6a081d71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 1 monclient: found mon.a 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a081d7a80 con 0x7f6a080b4700 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f69feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:40:52.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 10 monclient: _renew_subs 2024-03-25T19:40:52.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a08138960 con 0x7f6a080b4700 2024-03-25T19:40:52.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:52.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 10 monclient: _renew_subs 2024-03-25T19:40:52.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a08138600 con 0x7f6a080b4700 2024-03-25T19:40:52.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.423+0000 7f6a10552640 1 librados: init done 2024-03-25T19:40:52.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.424+0000 7f6a10552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:52.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.424+0000 7f6a10552640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:52.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.424+0000 7f6a10552640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.424+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0804f730 con 0x7f6a080b4700 2024-03-25T19:40:52.475 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.476+0000 7f123ec3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:52.477 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.478+0000 7fc741237040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:52.519 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.520+0000 7f123ec3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:52.525 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.525+0000 7fc741237040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:52.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.526+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69f8001210 con 0x7f6a080b4700 2024-03-25T19:40:52.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.526+0000 7f69feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:52.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.526+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f69f8003a40 con 0x7f6a080b4700 2024-03-25T19:40:52.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.526+0000 7f69feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.526+0000 7f69feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.526+0000 7f69feffd640 10 monclient: dump: 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.526+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f69f8003f30 con 0x7f6a080b4700 2024-03-25T19:40:52.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.526+0000 7f6a0caeb640 4 set_mon_vals no callback set 2024-03-25T19:40:52.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.527+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2550059950 0 0) 0x7f69f8005500 con 0x7f6a080b4700 2024-03-25T19:40:52.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.527+0000 7f69feffd640 10 client.4125.objecter ms_dispatch 0x7f6a08000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:40:52.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.527+0000 7f69feffd640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:40:52.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.527+0000 7f69feffd640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:40:52.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.527+0000 7f69feffd640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:40:52.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.527+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f69f8005810 con 0x7f6a080b4700 2024-03-25T19:40:52.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.528+0000 7f69feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:52.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.528+0000 7f69feffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:52.579 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.580+0000 7fc741237040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:52.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.601+0000 7f6a10552640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8dacde5d-1a0c-4aca-a176-862be9c3a92f", "id": 1}] 2024-03-25T19:40:52.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.601+0000 7f6a10552640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8dacde5d-1a0c-4aca-a176-862be9c3a92f", "id": 1}] 2024-03-25T19:40:52.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.601+0000 7f6a10552640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.601+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8dacde5d-1a0c-4aca-a176-862be9c3a92f", "id": 1} v 0) v1 -- 0x7f6a08069f30 con 0x7f6a080b4700 2024-03-25T19:40:52.631 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:52.631+0000 7fc741237040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:52.641 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.642+0000 7f123ec3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:52.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.667+0000 7f69feffd640 1 -- 172.21.15.53:0/3469406239 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8dacde5d-1a0c-4aca-a176-862be9c3a92f", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2732848768 0 940483276) 0x7f69f8005a50 con 0x7f6a080b4700 2024-03-25T19:40:52.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.667+0000 7f69feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8dacde5d-1a0c-4aca-a176-862be9c3a92f", "id": 1}] 2024-03-25T19:40:52.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.667+0000 7f69feffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:52.667 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-25T19:40:52.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 10 librados: watch_flush enter 2024-03-25T19:40:52.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 10 librados: watch_flush exit 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 10 monclient: shutdown 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 >> v1:172.21.15.18:6789/0 conn(0x7f6a080b4700 legacy=0x7f6a0813bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 shutdown_connections 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.668+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 >> 172.21.15.53:0/3469406239 conn(0x7f6a080668e0 msgr2=0x7f6a080b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.669+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 shutdown_connections 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.669+0000 7f6a10552640 1 -- 172.21.15.53:0/3469406239 wait complete. 2024-03-25T19:40:52.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.669+0000 7f6a10552640 1 librados: shutdown 2024-03-25T19:40:52.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 4095cdd1-ecc3-47d3-9923-7444cab97271 2 2024-03-25T19:40:52.716 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.716+0000 7f123ec3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:52.766 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.767+0000 7f123ec3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:52.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.817+0000 7fd77b16d640 10 monclient: get_monmap_and_config 2024-03-25T19:40:52.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.817+0000 7fd77b16d640 10 monclient: build_initial_monmap 2024-03-25T19:40:52.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.817+0000 7fd77b16d640 10 monclient: monmap: 2024-03-25T19:40:52.816 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:52.817937+0000 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:52.817937+0000 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:52.817 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.817+0000 7fd77b16d640 1 Processor -- start 2024-03-25T19:40:52.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.817+0000 7fd77b16d640 1 -- start start 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.817+0000 7fd77b16d640 10 monclient: init 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7740b4700 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 10 monclient(hunting): picked mon.noname-b con 0x7fd774142a20 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7740a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7740a9d60 con 0x7fd7740a96b0 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77413ff10 con 0x7fd7740b4700 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7740aa1f0 con 0x7fd774142a20 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd77b16d640 10 monclient(hunting): authenticate will time out at 4886.971171s 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd778f09640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd7740b4700 0x7fd774142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42626/0 (socket says 172.21.15.53:42626) 2024-03-25T19:40:52.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.818+0000 7fd778f09640 1 -- 172.21.15.53:0/1230026793 learned_addr learned my addr 172.21.15.53:0/1230026793 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:52.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807467300 0 0) 0x7fd77413ff10 con 0x7fd7740b4700 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd764003520 con 0x7fd7740b4700 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886240669 0 0) 0x7fd7740aa1f0 con 0x7fd774142a20 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:40:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.821+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77413ff10 con 0x7fd774142a20 2024-03-25T19:40:52.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.822+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327164349 0 0) 0x7fd7740a9d60 con 0x7fd7740a96b0 2024-03-25T19:40:52.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.822+0000 7fd7737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.822+0000 7fd7737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.822+0000 7fd7737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.822+0000 7fd7737fe640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:40:52.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.822+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7740aa1f0 con 0x7fd7740a96b0 2024-03-25T19:40:52.822 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.822+0000 7f123ec3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:52.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197126092 0 0) 0x7fd764003520 con 0x7fd7740b4700 2024-03-25T19:40:52.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 >> v1:172.21.15.18:6790/0 conn(0x7fd774142a20 legacy=0x7fd77413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 >> v1:172.21.15.18:6789/0 conn(0x7fd7740a96b0 legacy=0x7fd77413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7740b4ae0 con 0x7fd7740b4700 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fd768001690 con 0x7fd7740b4700 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.824+0000 7fd7737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.825+0000 7fd7737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.825+0000 7fd7737fe640 10 monclient: dump: 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.825 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.826 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.826 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.826 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.826 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.825+0000 7fd77b16d640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:40:52.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.825+0000 7fd77b16d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.852+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7680017f0 con 0x7fd7740b4700 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.852+0000 7fd7737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.852+0000 7fd7737fe640 1 -- 172.21.15.53:0/1230026793 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fd768001c00 con 0x7fd7740b4700 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.852+0000 7fd7737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.852+0000 7fd7737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.852+0000 7fd7737fe640 10 monclient: dump: 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.853 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.852+0000 7fd77b16d640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.852+0000 7fd77b16d640 4 set_mon_vals no callback set 2024-03-25T19:40:52.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.853+0000 7fd77b16d640 10 monclient: shutdown 2024-03-25T19:40:52.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.853+0000 7fd77b16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.853+0000 7fd77b16d640 1 -- 172.21.15.53:0/1230026793 >> v1:172.21.15.53:6789/0 conn(0x7fd7740b4700 legacy=0x7fd774142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.853+0000 7fd77b16d640 1 -- 172.21.15.53:0/1230026793 shutdown_connections 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.853+0000 7fd77b16d640 1 -- 172.21.15.53:0/1230026793 >> 172.21.15.53:0/1230026793 conn(0x7fd7740668e0 msgr2=0x7fd774066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.853+0000 7fd77b16d640 1 -- 172.21.15.53:0/1230026793 shutdown_connections 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.853+0000 7fd77b16d640 1 -- 172.21.15.53:0/1230026793 wait complete. 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 10 monclient: build_initial_monmap 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 10 monclient: monmap: 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 1 librados: starting msgr at 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 1 librados: starting objecter 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 1 Processor -- start 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 1 -- start start 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 1 librados: setting wanted keys 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 1 librados: calling monclient init 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.854+0000 7fd77b16d640 10 monclient: init 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7740a96b0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:52.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7740b4700 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 10 monclient(hunting): picked mon.noname-b con 0x7fd774142a20 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77413a440 con 0x7fd7740b4700 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7740ad0c0 con 0x7fd774142a20 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7740ad310 con 0x7fd7740a96b0 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.855+0000 7fd77b16d640 10 monclient(hunting): authenticate will time out at 4887.008471s 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.856+0000 7fd778f09640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fd7740a96b0 0x7fd7740ada10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.53:49610/0 (socket says 172.21.15.53:49610) 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.856+0000 7fd778f09640 1 -- 172.21.15.53:0/3553685857 learned_addr learned my addr 172.21.15.53:0/3553685857 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.856+0000 7fd7717fa640 10 client.?.objecter ms_handle_connect 0x7fd7740a96b0 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.856+0000 7fd7717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.856+0000 7fd7717fa640 10 client.?.objecter ms_handle_connect 0x7fd7740b4700 2024-03-25T19:40:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.856+0000 7fd7717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:52.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.857+0000 7fd7717fa640 10 client.?.objecter ms_handle_connect 0x7fd774142a20 2024-03-25T19:40:52.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.857+0000 7fd7717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:52.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998233101 0 0) 0x7fd7740ad310 con 0x7fd7740a96b0 2024-03-25T19:40:52.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd758004eb0 con 0x7fd7740a96b0 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889147006 0 0) 0x7fd77413ff10 con 0x7fd7740b4700 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.858+0000 7fd7717fa640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.859+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7740ad310 con 0x7fd7740b4700 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.859+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502978149 0 0) 0x7fd7740ad0c0 con 0x7fd774142a20 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.859+0000 7fd7717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.859+0000 7fd7717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:52.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.859+0000 7fd7717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.859+0000 7fd7717fa640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:40:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.859+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77413ff10 con 0x7fd774142a20 2024-03-25T19:40:52.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 249692564 0 0) 0x7fd7740ad310 con 0x7fd7740b4700 2024-03-25T19:40:52.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7740ad0c0 con 0x7fd7740b4700 2024-03-25T19:40:52.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fd7600017d0 con 0x7fd7740b4700 2024-03-25T19:40:52.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 10 monclient(hunting): dump: 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:52.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1700972628 0 0) 0x7fd758004eb0 con 0x7fd7740a96b0 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7740ad310 con 0x7fd7740a96b0 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fd768001bc0 con 0x7fd7740a96b0 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.863+0000 7fd7717fa640 10 monclient(hunting): dump: 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.864+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3131521223 0 0) 0x7fd77413ff10 con 0x7fd774142a20 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.864+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd758004eb0 con 0x7fd774142a20 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.864+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fd76c001bd0 con 0x7fd774142a20 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.864+0000 7fd7717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.864+0000 7fd7717fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.864+0000 7fd7717fa640 10 monclient(hunting): dump: 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.866 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.867 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.867 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.867+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2535182678 0 0) 0x7fd7740ad0c0 con 0x7fd7740b4700 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.867+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 >> v1:172.21.15.18:6790/0 conn(0x7fd7740a96b0 legacy=0x7fd7740ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.867+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 >> v1:172.21.15.53:6789/0 conn(0x7fd774142a20 legacy=0x7fd774139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.867+0000 7fd7717fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.867+0000 7fd7717fa640 1 monclient: found mon.a 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.867+0000 7fd7717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.867+0000 7fd7717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.867+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7740ad5f0 con 0x7fd7740b4700 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd7717fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd7717fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd7717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 5 monclient: authenticate success, global_id 4131 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 10 monclient: _renew_subs 2024-03-25T19:40:52.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7740ae120 con 0x7fd7740b4700 2024-03-25T19:40:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 10 monclient: _renew_subs 2024-03-25T19:40:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7740abd30 con 0x7fd7740b4700 2024-03-25T19:40:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.868+0000 7fd77b16d640 1 librados: init done 2024-03-25T19:40:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.869+0000 7fd77b16d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.869+0000 7fd77b16d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:52.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.869+0000 7fd77b16d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:52.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.869+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77404f730 con 0x7fd7740b4700 2024-03-25T19:40:52.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd760001c40 con 0x7fd7740b4700 2024-03-25T19:40:52.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:52.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fd760003790 con 0x7fd7740b4700 2024-03-25T19:40:52.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:52.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:52.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 10 monclient: dump: 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd760003c80 con 0x7fd7740b4700 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd772ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2039918415 0 0) 0x7fd7600051f0 con 0x7fd7740b4700 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 10 client.4131.objecter ms_dispatch 0x7fd774000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:40:52.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.873+0000 7fd7717fa640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T19:40:52.874 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:52.873+0000 7f123ec3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:52.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.973+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd760004ec0 con 0x7fd7740b4700 2024-03-25T19:40:52.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.973+0000 7fd7717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:52.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:52.973+0000 7fd7717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:53.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.047+0000 7fd77b16d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4095cdd1-ecc3-47d3-9923-7444cab97271", "id": 2}] 2024-03-25T19:40:53.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.047+0000 7fd77b16d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4095cdd1-ecc3-47d3-9923-7444cab97271", "id": 2}] 2024-03-25T19:40:53.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.047+0000 7fd77b16d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.047+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4095cdd1-ecc3-47d3-9923-7444cab97271", "id": 2} v 0) v1 -- 0x7fd7740b7190 con 0x7fd7740b4700 2024-03-25T19:40:53.054 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.055+0000 7fc741237040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:40:53.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.078+0000 7fd7717fa640 1 -- 172.21.15.53:0/3553685857 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4095cdd1-ecc3-47d3-9923-7444cab97271", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2901553202 0 216599125) 0x7fd760003f30 con 0x7fd7740b4700 2024-03-25T19:40:53.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.078+0000 7fd7717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4095cdd1-ecc3-47d3-9923-7444cab97271", "id": 2}] 2024-03-25T19:40:53.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.078+0000 7fd7717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:53.078 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-03-25T19:40:53.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.079+0000 7fd77b16d640 10 librados: watch_flush enter 2024-03-25T19:40:53.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.079+0000 7fd77b16d640 10 librados: watch_flush exit 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.079+0000 7fd77b16d640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.079+0000 7fd77b16d640 10 client.4131.objecter successfully canceled tick 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.079+0000 7fd77b16d640 10 monclient: shutdown 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.079+0000 7fd77b16d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.079+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 >> v1:172.21.15.18:6789/0 conn(0x7fd7740b4700 legacy=0x7fd774138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.080+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 shutdown_connections 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.080+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 >> 172.21.15.53:0/3553685857 conn(0x7fd7740668e0 msgr2=0x7fd7741364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.080+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 shutdown_connections 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.080+0000 7fd77b16d640 1 -- 172.21.15.53:0/3553685857 wait complete. 2024-03-25T19:40:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.080+0000 7fd77b16d640 1 librados: shutdown 2024-03-25T19:40:53.090 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new f3ca72b0-c0c2-4d82-a95d-814c63529db1 3 2024-03-25T19:40:53.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.191+0000 7f26e46b3640 10 monclient: get_monmap_and_config 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.191+0000 7f26e46b3640 10 monclient: build_initial_monmap 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.191+0000 7f26e46b3640 10 monclient: monmap: 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:53.192241+0000 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:53.192241+0000 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:53.191 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.191+0000 7f26e46b3640 1 Processor -- start 2024-03-25T19:40:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.191+0000 7f26e46b3640 1 -- start start 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.191+0000 7f26e46b3640 10 monclient: init 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 10 monclient(hunting): picked mon.noname-b con 0x7f26dc0b4700 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 10 monclient(hunting): picked mon.noname-c con 0x7f26dc142a20 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 10 monclient(hunting): picked mon.noname-a con 0x7f26dc0a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0a9d60 con 0x7f26dc0a96b0 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc13ff10 con 0x7f26dc142a20 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0aa1f0 con 0x7f26dc0b4700 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.192+0000 7f26e46b3640 10 monclient(hunting): authenticate will time out at 4887.345466s 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.193+0000 7f26e1c4e640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f26dc142a20 0x7f26dc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42648/0 (socket says 172.21.15.53:42648) 2024-03-25T19:40:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.193+0000 7f26e1c4e640 1 -- 172.21.15.53:0/1758432645 learned_addr learned my addr 172.21.15.53:0/1758432645 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137894481 0 0) 0x7f26dc13ff10 con 0x7f26dc142a20 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8003520 con 0x7f26dc142a20 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719010694 0 0) 0x7f26dc0aa1f0 con 0x7f26dc0b4700 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc13ff10 con 0x7f26dc0b4700 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920818265 0 0) 0x7f26dc0a9d60 con 0x7f26dc0a96b0 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:40:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.195+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc0aa1f0 con 0x7f26dc0a96b0 2024-03-25T19:40:53.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1700566942 0 0) 0x7f26c8003520 con 0x7f26dc142a20 2024-03-25T19:40:53.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 >> v1:172.21.15.18:6790/0 conn(0x7f26dc0b4700 legacy=0x7f26dc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 >> v1:172.21.15.18:6789/0 conn(0x7f26dc0a96b0 legacy=0x7f26dc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 10 monclient: _finish_hunting 0 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 1 monclient: found mon.noname-c 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0b4ae0 con 0x7f26dc142a20 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 10 monclient: _finish_auth 0 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 10 monclient: _check_auth_tickets 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f26cc001690 con 0x7f26dc142a20 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e144d640 10 monclient: dump: 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.200 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e46b3640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:40:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.198+0000 7f26e46b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26cc0017f0 con 0x7f26dc142a20 2024-03-25T19:40:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e144d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e144d640 1 -- 172.21.15.53:0/1758432645 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f26cc001c00 con 0x7f26dc142a20 2024-03-25T19:40:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e144d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 4 set_mon_vals no callback set 2024-03-25T19:40:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e144d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e144d640 10 monclient: dump: 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 10 monclient: shutdown 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 1 -- 172.21.15.53:0/1758432645 >> v1:172.21.15.53:6789/0 conn(0x7f26dc142a20 legacy=0x7f26dc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 1 -- 172.21.15.53:0/1758432645 shutdown_connections 2024-03-25T19:40:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 1 -- 172.21.15.53:0/1758432645 >> 172.21.15.53:0/1758432645 conn(0x7f26dc0668e0 msgr2=0x7f26dc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 1 -- 172.21.15.53:0/1758432645 shutdown_connections 2024-03-25T19:40:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.201+0000 7f26e46b3640 1 -- 172.21.15.53:0/1758432645 wait complete. 2024-03-25T19:40:53.203 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.202+0000 7fc741237040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.202+0000 7f26e46b3640 10 monclient: build_initial_monmap 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.202+0000 7f26e46b3640 10 monclient: monmap: 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:53.206 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.202+0000 7f26e46b3640 1 librados: starting msgr at 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.202+0000 7f26e46b3640 1 librados: starting objecter 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.202+0000 7f26e46b3640 1 Processor -- start 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 1 -- start start 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 1 librados: setting wanted keys 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 1 librados: calling monclient init 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 10 monclient: init 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 10 monclient(hunting): picked mon.noname-c con 0x7f26dc0a96b0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 10 monclient(hunting): picked mon.noname-b con 0x7f26dc0b4700 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 10 monclient(hunting): picked mon.noname-a con 0x7f26dc142a20 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0ade40 con 0x7f26dc142a20 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0ae0b0 con 0x7f26dc0b4700 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26dc0ae390 con 0x7f26dc0a96b0 2024-03-25T19:40:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e46b3640 10 monclient(hunting): authenticate will time out at 4887.356343s 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e1c4e640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f26dc0b4700 0x7f26dc0ae7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42658/0 (socket says 172.21.15.53:42658) 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.203+0000 7f26e1c4e640 1 -- 172.21.15.53:0/1562157478 learned_addr learned my addr 172.21.15.53:0/1562157478 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.204+0000 7f26d2ffd640 10 client.?.objecter ms_handle_connect 0x7f26dc0b4700 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.204+0000 7f26d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.204+0000 7f26d2ffd640 10 client.?.objecter ms_handle_connect 0x7f26dc0a96b0 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.204+0000 7f26d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.204+0000 7f26d2ffd640 10 client.?.objecter ms_handle_connect 0x7f26dc142a20 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.204+0000 7f26d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370101102 0 0) 0x7f26dc0ae390 con 0x7f26dc0a96b0 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c0004eb0 con 0x7f26dc0a96b0 2024-03-25T19:40:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395223981 0 0) 0x7f26dc0ae0b0 con 0x7f26dc0b4700 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc0ae390 con 0x7f26dc0b4700 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171086785 0 0) 0x7f26dc0ade40 con 0x7f26dc142a20 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:40:53.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.206+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26dc0ae0b0 con 0x7f26dc142a20 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 578878463 0 0) 0x7f26dc0ae390 con 0x7f26dc0b4700 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26dc0ade40 con 0x7f26dc0b4700 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f26cc001b40 con 0x7f26dc0b4700 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.211 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2463572473 0 0) 0x7f26dc0ae0b0 con 0x7f26dc142a20 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26dc0ae390 con 0x7f26dc142a20 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1539436391 0 0) 0x7f26c0004eb0 con 0x7f26dc0a96b0 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26dc0ae0b0 con 0x7f26dc0a96b0 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f26d8001970 con 0x7f26dc0a96b0 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.212 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f26d4001b90 con 0x7f26dc142a20 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.211+0000 7f26d2ffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.213 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.214 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.214 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.214 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 624174146 0 0) 0x7f26dc0ae390 con 0x7f26dc142a20 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 >> v1:172.21.15.18:6790/0 conn(0x7f26dc0a96b0 legacy=0x7f26dc13aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 >> v1:172.21.15.53:6789/0 conn(0x7f26dc0b4700 legacy=0x7f26dc0ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 1 monclient: found mon.a 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc0ad060 con 0x7f26dc142a20 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:53.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 5 monclient: authenticate success, global_id 4137 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 10 monclient: _renew_subs 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc0abcc0 con 0x7f26dc142a20 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 10 monclient: _renew_subs 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc0ac960 con 0x7f26dc142a20 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.214+0000 7f26e46b3640 1 librados: init done 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.215+0000 7f26e46b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.215+0000 7f26e46b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.215+0000 7f26e46b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.215+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc04f730 con 0x7f26dc142a20 2024-03-25T19:40:53.274 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:53.274+0000 7f123ec3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.319+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26d4002020 con 0x7f26dc142a20 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.319+0000 7f26d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.319+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f26d4003b30 con 0x7f26dc142a20 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.319+0000 7f26d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.319+0000 7f26d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.319+0000 7f26d2ffd640 10 monclient: dump: 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.319+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f26d4004020 con 0x7f26dc142a20 2024-03-25T19:40:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.319+0000 7f26e0c4c640 4 set_mon_vals no callback set 2024-03-25T19:40:53.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.320+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 22286737 0 0) 0x7f26d40056f0 con 0x7f26dc142a20 2024-03-25T19:40:53.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.320+0000 7f26d2ffd640 10 client.4137.objecter ms_dispatch 0x7f26dc000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:40:53.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.320+0000 7f26d2ffd640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:40:53.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.320+0000 7f26d2ffd640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:40:53.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.320+0000 7f26d2ffd640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T19:40:53.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.320+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f26d4005260 con 0x7f26dc142a20 2024-03-25T19:40:53.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.320+0000 7f26d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.320+0000 7f26d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:53.336 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.337+0000 7fc741237040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:40:53.384 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.384+0000 7fc741237040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:40:53.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.394+0000 7f26e46b3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f3ca72b0-c0c2-4d82-a95d-814c63529db1", "id": 3}] 2024-03-25T19:40:53.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.394+0000 7f26e46b3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f3ca72b0-c0c2-4d82-a95d-814c63529db1", "id": 3}] 2024-03-25T19:40:53.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.394+0000 7f26e46b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.394+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f3ca72b0-c0c2-4d82-a95d-814c63529db1", "id": 3} v 0) v1 -- 0x7f26dc13e5d0 con 0x7f26dc142a20 2024-03-25T19:40:53.415 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:53.415+0000 7f123ec3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:40:53.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.428+0000 7f26d2ffd640 1 -- 172.21.15.53:0/1562157478 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f3ca72b0-c0c2-4d82-a95d-814c63529db1", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1449464785 0 525046306) 0x7f26d4005550 con 0x7f26dc142a20 2024-03-25T19:40:53.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.428+0000 7f26d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f3ca72b0-c0c2-4d82-a95d-814c63529db1", "id": 3}] 2024-03-25T19:40:53.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.428+0000 7f26d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:53.428 INFO:teuthology.orchestra.run.smithi053.stdout:3 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 10 librados: watch_flush enter 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 10 librados: watch_flush exit 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 10 client.4137.objecter successfully canceled tick 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 10 monclient: shutdown 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 >> v1:172.21.15.18:6789/0 conn(0x7f26dc142a20 legacy=0x7f26dc1e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 shutdown_connections 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 >> 172.21.15.53:0/1562157478 conn(0x7f26dc0668e0 msgr2=0x7f26dc0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 shutdown_connections 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 1 -- 172.21.15.53:0/1562157478 wait complete. 2024-03-25T19:40:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.429+0000 7f26e46b3640 1 librados: shutdown 2024-03-25T19:40:53.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 7d48b076-42ea-4b65-81d9-8fe8cdc9b352 4 2024-03-25T19:40:53.504 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.505+0000 7fc741237040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.540+0000 7f3ae4fa3640 10 monclient: get_monmap_and_config 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.540+0000 7f3ae4fa3640 10 monclient: build_initial_monmap 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.540+0000 7f3ae4fa3640 10 monclient: monmap: 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:53.541693+0000 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:53.541693+0000 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:53.541 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:53.541 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.541+0000 7f3ae4fa3640 1 Processor -- start 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.541+0000 7f3ae4fa3640 1 -- start start 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.541+0000 7f3ae4fa3640 10 monclient: init 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f3ae00b4700 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f3ae0142a20 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f3ae00a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae00a9d60 con 0x7f3ae00a96b0 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae013ff10 con 0x7f3ae00b4700 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae00aa1f0 con 0x7f3ae0142a20 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ae4fa3640 10 monclient(hunting): authenticate will time out at 4887.694924s 2024-03-25T19:40:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ade59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3ae00b4700 0x7f3ae0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42668/0 (socket says 172.21.15.53:42668) 2024-03-25T19:40:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.542+0000 7f3ade59c640 1 -- 172.21.15.53:0/3418797035 learned_addr learned my addr 172.21.15.53:0/3418797035 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415942477 0 0) 0x7f3ae00aa1f0 con 0x7f3ae0142a20 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac8003520 con 0x7f3ae0142a20 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074784970 0 0) 0x7f3ae00a9d60 con 0x7f3ae00a96b0 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:40:53.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.545+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae00aa1f0 con 0x7f3ae00a96b0 2024-03-25T19:40:53.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.546+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203637064 0 0) 0x7f3ae013ff10 con 0x7f3ae00b4700 2024-03-25T19:40:53.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.546+0000 7f3add59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.546+0000 7f3add59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.546+0000 7f3add59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.546+0000 7f3add59a640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:40:53.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.546+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae00a9d60 con 0x7f3ae00b4700 2024-03-25T19:40:53.547 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:53.547+0000 7f123ec3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:40:53.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 403084441 0 0) 0x7f3ac8003520 con 0x7f3ae0142a20 2024-03-25T19:40:53.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 >> v1:172.21.15.53:6789/0 conn(0x7f3ae00b4700 legacy=0x7f3ae0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 >> v1:172.21.15.18:6789/0 conn(0x7f3ae00a96b0 legacy=0x7f3ae013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 10 monclient: _finish_hunting 0 2024-03-25T19:40:53.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 1 monclient: found mon.noname-b 2024-03-25T19:40:53.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:53.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00b4ae0 con 0x7f3ae0142a20 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 10 monclient: _finish_auth 0 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 10 monclient: _check_auth_tickets 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3ad4001690 con 0x7f3ae0142a20 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3add59a640 10 monclient: dump: 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.550 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.550 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.550 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3ae4fa3640 5 monclient: authenticate success, global_id 4133 2024-03-25T19:40:53.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.548+0000 7f3ae4fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:53.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.575+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ad40017f0 con 0x7f3ae0142a20 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.575+0000 7f3add59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.575+0000 7f3add59a640 1 -- 172.21.15.53:0/3418797035 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3ad4001c00 con 0x7f3ae0142a20 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.575+0000 7f3add59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.575+0000 7f3add59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.575+0000 7f3add59a640 10 monclient: dump: 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.576 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.575+0000 7f3ae4fa3640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.575+0000 7f3ae4fa3640 4 set_mon_vals no callback set 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.576+0000 7f3ae4fa3640 10 monclient: shutdown 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.576+0000 7f3ae4fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.576+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/3418797035 >> v1:172.21.15.18:6790/0 conn(0x7f3ae0142a20 legacy=0x7f3ae013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.576+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/3418797035 shutdown_connections 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.576+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/3418797035 >> 172.21.15.53:0/3418797035 conn(0x7f3ae00668e0 msgr2=0x7f3ae0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.576+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/3418797035 shutdown_connections 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.576+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/3418797035 wait complete. 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 10 monclient: build_initial_monmap 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 10 monclient: monmap: 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-03-25T19:40:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 1 librados: starting msgr at 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 1 librados: starting objecter 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 1 Processor -- start 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 1 -- start start 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 1 librados: setting wanted keys 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 1 librados: calling monclient init 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.577+0000 7f3ae4fa3640 10 monclient: init 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:53.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f3ae00a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f3ae00b4700 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f3ae0142a20 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae0131a00 con 0x7f3ae00a96b0 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae00ada30 con 0x7f3ae0142a20 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae00af460 con 0x7f3ae00b4700 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3ae4fa3640 10 monclient(hunting): authenticate will time out at 4887.730901s 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3addd9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3ae00b4700 0x7f3ae0130be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.53:49656/0 (socket says 172.21.15.53:49656) 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3addd9b640 1 -- 172.21.15.53:0/1801611544 learned_addr learned my addr 172.21.15.53:0/1801611544 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3aceffd640 10 client.?.objecter ms_handle_connect 0x7f3ae00b4700 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3aceffd640 10 client.?.objecter ms_handle_connect 0x7f3ae0142a20 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.578+0000 7f3aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.579+0000 7f3aceffd640 10 client.?.objecter ms_handle_connect 0x7f3ae00a96b0 2024-03-25T19:40:53.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.579+0000 7f3aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660023049 0 0) 0x7f3ae00af460 con 0x7f3ae00b4700 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac0004eb0 con 0x7f3ae00b4700 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488934186 0 0) 0x7f3ae00ada30 con 0x7f3ae0142a20 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.580+0000 7f3aceffd640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.581+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae00af460 con 0x7f3ae0142a20 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.581+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027519707 0 0) 0x7f3ae0131a00 con 0x7f3ae00a96b0 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.581+0000 7f3aceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.581+0000 7f3aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.581+0000 7f3aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.581+0000 7f3aceffd640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:40:53.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.581+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae00ada30 con 0x7f3ae00a96b0 2024-03-25T19:40:53.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355707920 0 0) 0x7f3ac0004eb0 con 0x7f3ae00b4700 2024-03-25T19:40:53.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ae0131a00 con 0x7f3ae00b4700 2024-03-25T19:40:53.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3ad4001bc0 con 0x7f3ae00b4700 2024-03-25T19:40:53.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3921040118 0 0) 0x7f3ae00ada30 con 0x7f3ae00a96b0 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ac0004eb0 con 0x7f3ae00a96b0 2024-03-25T19:40:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3ad0001830 con 0x7f3ae00a96b0 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1108570438 0 0) 0x7f3ae00af460 con 0x7f3ae0142a20 2024-03-25T19:40:53.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.586+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ae00ada30 con 0x7f3ae0142a20 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.587+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3ad8001b50 con 0x7f3ae0142a20 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.587+0000 7f3aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.587+0000 7f3aceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.587+0000 7f3aceffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.589 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 582296123 0 0) 0x7f3ac0004eb0 con 0x7f3ae00a96b0 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 >> v1:172.21.15.18:6790/0 conn(0x7f3ae00b4700 legacy=0x7f3ae0130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 >> v1:172.21.15.53:6789/0 conn(0x7f3ae0142a20 legacy=0x7f3ae01312f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 1 monclient: found mon.a 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00af740 con 0x7f3ae00a96b0 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3aceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 5 monclient: authenticate success, global_id 4143 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 10 monclient: _renew_subs 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae00b0270 con 0x7f3ae00a96b0 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 10 monclient: _renew_subs 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae00ad250 con 0x7f3ae00a96b0 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.590+0000 7f3ae4fa3640 1 librados: init done 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.591+0000 7f3ae4fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.591+0000 7f3ae4fa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.591+0000 7f3ae4fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.591+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae00adbc0 con 0x7f3ae00a96b0 2024-03-25T19:40:53.595 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:53.596+0000 7f123ec3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:40:53.607 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.608+0000 7fc741237040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:40:53.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.699+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ad0001d40 con 0x7f3ae00a96b0 2024-03-25T19:40:53.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.699+0000 7f3aceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.699+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3ad0004820 con 0x7f3ae00a96b0 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.699+0000 7f3aceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.699+0000 7f3aceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.699+0000 7f3aceffd640 10 monclient: dump: 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.699+0000 7f3adcd99640 4 set_mon_vals no callback set 2024-03-25T19:40:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.699+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3ad0004dd0 con 0x7f3ae00a96b0 2024-03-25T19:40:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.700+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3620325904 0 0) 0x7f3ad00064a0 con 0x7f3ae00a96b0 2024-03-25T19:40:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.700+0000 7f3aceffd640 10 client.4143.objecter ms_dispatch 0x7f3ae0000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:40:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.700+0000 7f3aceffd640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:40:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.700+0000 7f3aceffd640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:40:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.700+0000 7f3aceffd640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T19:40:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.700+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3ad00067f0 con 0x7f3ae00a96b0 2024-03-25T19:40:53.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.700+0000 7f3aceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.700+0000 7f3aceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:53.712 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:53.712+0000 7f123ec3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:53.779 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.779+0000 7fc741237040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:53.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.792+0000 7f3ae4fa3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7d48b076-42ea-4b65-81d9-8fe8cdc9b352", "id": 4}] 2024-03-25T19:40:53.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.792+0000 7f3ae4fa3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7d48b076-42ea-4b65-81d9-8fe8cdc9b352", "id": 4}] 2024-03-25T19:40:53.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.792+0000 7f3ae4fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:53.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.792+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7d48b076-42ea-4b65-81d9-8fe8cdc9b352", "id": 4} v 0) v1 -- 0x7f3ae004f730 con 0x7f3ae00a96b0 2024-03-25T19:40:53.812 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:53.813+0000 7f123ec3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:40:53.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.827+0000 7f3aceffd640 1 -- 172.21.15.53:0/1801611544 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7d48b076-42ea-4b65-81d9-8fe8cdc9b352", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3426009255 0 1697012583) 0x7f3ad0006060 con 0x7f3ae00a96b0 2024-03-25T19:40:53.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.827+0000 7f3aceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7d48b076-42ea-4b65-81d9-8fe8cdc9b352", "id": 4}] 2024-03-25T19:40:53.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.827+0000 7f3aceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:4 2024-03-25T19:40:53.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 10 librados: watch_flush enter 2024-03-25T19:40:53.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 10 librados: watch_flush exit 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 10 client.4143.objecter successfully canceled tick 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 10 monclient: shutdown 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 >> v1:172.21.15.18:6789/0 conn(0x7f3ae00a96b0 legacy=0x7f3ae00afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 shutdown_connections 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 >> 172.21.15.53:0/1801611544 conn(0x7f3ae00668e0 msgr2=0x7f3ae00b6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 shutdown_connections 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.828+0000 7f3ae4fa3640 1 -- 172.21.15.53:0/1801611544 wait complete. 2024-03-25T19:40:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.829+0000 7f3ae4fa3640 1 librados: shutdown 2024-03-25T19:40:53.830 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.829+0000 7fc741237040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:53.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 5ca4d285-71b8-402d-b33c-496cbe7cf78f 5 2024-03-25T19:40:53.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.938+0000 7f38fcc6a640 10 monclient: get_monmap_and_config 2024-03-25T19:40:53.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.938+0000 7f38fcc6a640 10 monclient: build_initial_monmap 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.938+0000 7f38fcc6a640 10 monclient: monmap: 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:53.939761+0000 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:53.939761+0000 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.939+0000 7f38fcc6a640 1 Processor -- start 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.939+0000 7f38fcc6a640 1 -- start start 2024-03-25T19:40:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.939+0000 7f38fcc6a640 10 monclient: init 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 10 monclient(hunting): picked mon.noname-a con 0x7f38f80b4700 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 10 monclient(hunting): picked mon.noname-b con 0x7f38f8142a20 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 10 monclient(hunting): picked mon.noname-c con 0x7f38f80a96b0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80a9d60 con 0x7f38f80b4700 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f813ff10 con 0x7f38f80a96b0 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80aa1f0 con 0x7f38f8142a20 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38fcc6a640 10 monclient(hunting): authenticate will time out at 4888.092982s 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38f6d9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f38f80a96b0 0x7f38f813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42686/0 (socket says 172.21.15.53:42686) 2024-03-25T19:40:53.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.940+0000 7f38f6d9d640 1 -- 172.21.15.53:0/1185568813 learned_addr learned my addr 172.21.15.53:0/1185568813 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156796749 0 0) 0x7f38f813ff10 con 0x7f38f80a96b0 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e4003520 con 0x7f38f80a96b0 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901857854 0 0) 0x7f38f80a9d60 con 0x7f38f80b4700 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:40:53.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f813ff10 con 0x7f38f80b4700 2024-03-25T19:40:53.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57390654 0 0) 0x7f38f80aa1f0 con 0x7f38f8142a20 2024-03-25T19:40:53.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:40:53.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.943+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f80a9d60 con 0x7f38f8142a20 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1116909253 0 0) 0x7f38e4003520 con 0x7f38f80a96b0 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 >> v1:172.21.15.18:6790/0 conn(0x7f38f8142a20 legacy=0x7f38f813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 >> v1:172.21.15.18:6789/0 conn(0x7f38f80b4700 legacy=0x7f38f8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 1 monclient: found mon.noname-c 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f80b4ae0 con 0x7f38f80a96b0 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f38f0001a10 con 0x7f38f80a96b0 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38f5d9b640 10 monclient: dump: 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38fcc6a640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:40:53.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.946+0000 7f38fcc6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38f0001b70 con 0x7f38f80a96b0 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38f5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38f5d9b640 1 -- 172.21.15.53:0/1185568813 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f38f0001f80 con 0x7f38f80a96b0 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 4 set_mon_vals no callback set 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38f5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38f5d9b640 10 monclient: dump: 2024-03-25T19:40:53.949 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 10 monclient: shutdown 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 1 -- 172.21.15.53:0/1185568813 >> v1:172.21.15.53:6789/0 conn(0x7f38f80a96b0 legacy=0x7f38f813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 1 -- 172.21.15.53:0/1185568813 shutdown_connections 2024-03-25T19:40:53.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 1 -- 172.21.15.53:0/1185568813 >> 172.21.15.53:0/1185568813 conn(0x7f38f80668e0 msgr2=0x7f38f8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 1 -- 172.21.15.53:0/1185568813 shutdown_connections 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.948+0000 7f38fcc6a640 1 -- 172.21.15.53:0/1185568813 wait complete. 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.949+0000 7f38fcc6a640 10 monclient: build_initial_monmap 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.949+0000 7f38fcc6a640 10 monclient: monmap: 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.949+0000 7f38fcc6a640 1 librados: starting msgr at 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.949+0000 7f38fcc6a640 1 librados: starting objecter 2024-03-25T19:40:53.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.949+0000 7f38fcc6a640 1 Processor -- start 2024-03-25T19:40:53.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 1 -- start start 2024-03-25T19:40:53.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 1 librados: setting wanted keys 2024-03-25T19:40:53.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 1 librados: calling monclient init 2024-03-25T19:40:53.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 10 monclient: init 2024-03-25T19:40:53.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 10 monclient(hunting): picked mon.noname-c con 0x7f38f80a96b0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 10 monclient(hunting): picked mon.noname-b con 0x7f38f80b4700 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 10 monclient(hunting): picked mon.noname-a con 0x7f38f8142a20 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80add10 con 0x7f38f8142a20 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f80acb70 con 0x7f38f80b4700 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f814e7e0 con 0x7f38f80a96b0 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.950+0000 7f38fcc6a640 10 monclient(hunting): authenticate will time out at 4888.103433s 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.951+0000 7f38effff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f38f80b4700 0x7f38f8136060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42692/0 (socket says 172.21.15.53:42692) 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.951+0000 7f38effff640 1 -- 172.21.15.53:0/280891295 learned_addr learned my addr 172.21.15.53:0/280891295 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.951+0000 7f38eeffd640 10 client.?.objecter ms_handle_connect 0x7f38f80b4700 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.951+0000 7f38eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.951+0000 7f38eeffd640 10 client.?.objecter ms_handle_connect 0x7f38f80a96b0 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.951+0000 7f38eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058996236 0 0) 0x7f38f80acb70 con 0x7f38f80b4700 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d8004eb0 con 0x7f38f80b4700 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 client.?.objecter ms_handle_connect 0x7f38f8142a20 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184335097 0 0) 0x7f38f814e7e0 con 0x7f38f80a96b0 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:40:53.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.954+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f80acb70 con 0x7f38f80a96b0 2024-03-25T19:40:53.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.956+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715368934 0 0) 0x7f38f80add10 con 0x7f38f8142a20 2024-03-25T19:40:53.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.956+0000 7f38eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.956+0000 7f38eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.956+0000 7f38eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.956+0000 7f38eeffd640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:40:53.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.956+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f814e7e0 con 0x7f38f8142a20 2024-03-25T19:40:53.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.958+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578327396 0 0) 0x7f38d8004eb0 con 0x7f38f80b4700 2024-03-25T19:40:53.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.958+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38f80add10 con 0x7f38f80b4700 2024-03-25T19:40:53.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.958+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f38e0001970 con 0x7f38f80b4700 2024-03-25T19:40:53.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.958+0000 7f38eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.959+0000 7f38eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.959+0000 7f38eeffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.959+0000 7f38eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.960+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893550850 0 0) 0x7f38f80acb70 con 0x7f38f80a96b0 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.960+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d8004eb0 con 0x7f38f80a96b0 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.960+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f38e8001970 con 0x7f38f80a96b0 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.960+0000 7f38eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.960+0000 7f38eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.960+0000 7f38eeffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.961 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.961 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.961 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.961 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.961 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.961 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.961+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2609256815 0 0) 0x7f38f814e7e0 con 0x7f38f8142a20 2024-03-25T19:40:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.961+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38f80acb70 con 0x7f38f8142a20 2024-03-25T19:40:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.961+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f38f0001e80 con 0x7f38f8142a20 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.961+0000 7f38eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.961+0000 7f38eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.961+0000 7f38eeffd640 10 monclient(hunting): dump: 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2787360839 0 0) 0x7f38f80add10 con 0x7f38f80b4700 2024-03-25T19:40:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 >> v1:172.21.15.18:6790/0 conn(0x7f38f80a96b0 legacy=0x7f38f80ac370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 >> v1:172.21.15.18:6789/0 conn(0x7f38f8142a20 legacy=0x7f38f81daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 1 monclient: found mon.b 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f80ab760 con 0x7f38f80b4700 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 10 monclient: _renew_subs 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f8136770 con 0x7f38f80b4700 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 10 monclient: _renew_subs 2024-03-25T19:40:53.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:40:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f81db380 con 0x7f38f80b4700 2024-03-25T19:40:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.962+0000 7f38fcc6a640 1 librados: init done 2024-03-25T19:40:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.963+0000 7f38fcc6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.963+0000 7f38fcc6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.963+0000 7f38fcc6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:40:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:53.963+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f80ae230 con 0x7f38f80b4700 2024-03-25T19:40:53.989 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:53.990+0000 7f123ec3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:40:53.995 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:53.996+0000 7fc741237040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:40:54.034 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:54.035+0000 7f123ec3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:54.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.170+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38e0003610 con 0x7f38f80b4700 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.170+0000 7f38eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.170+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f38e0003880 con 0x7f38f80b4700 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.170+0000 7f38eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.170+0000 7f38eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.170+0000 7f38eeffd640 10 monclient: dump: 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:54.171 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.170+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f38e0003e30 con 0x7f38f80b4700 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.170+0000 7f38f559a640 4 set_mon_vals no callback set 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.171+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3517320692 0 0) 0x7f38e0006610 con 0x7f38f80b4700 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.171+0000 7f38eeffd640 10 client.4144.objecter ms_dispatch 0x7f38f8000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.171+0000 7f38eeffd640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.171+0000 7f38eeffd640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.171+0000 7f38eeffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.171+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f38e0006990 con 0x7f38f80b4700 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.171+0000 7f38eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.171+0000 7f38eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:54.195 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:54.196+0000 7f123ec3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:40:54.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.244+0000 7f38fcc6a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5ca4d285-71b8-402d-b33c-496cbe7cf78f", "id": 5}] 2024-03-25T19:40:54.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.244+0000 7f38fcc6a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5ca4d285-71b8-402d-b33c-496cbe7cf78f", "id": 5}] 2024-03-25T19:40:54.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.244+0000 7f38fcc6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:40:54.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.244+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5ca4d285-71b8-402d-b33c-496cbe7cf78f", "id": 5} v 0) v1 -- 0x7f38f80acd00 con 0x7f38f80b4700 2024-03-25T19:40:54.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.344+0000 7f38eeffd640 1 -- 172.21.15.53:0/280891295 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5ca4d285-71b8-402d-b33c-496cbe7cf78f", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2012083688 0 1988412176) 0x7f38e0006bd0 con 0x7f38f80b4700 2024-03-25T19:40:54.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.344+0000 7f38eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5ca4d285-71b8-402d-b33c-496cbe7cf78f", "id": 5}] 2024-03-25T19:40:54.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.344+0000 7f38eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:54.344 INFO:teuthology.orchestra.run.smithi053.stdout:5 2024-03-25T19:40:54.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.344+0000 7f38fcc6a640 10 librados: watch_flush enter 2024-03-25T19:40:54.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 10 librados: watch_flush exit 2024-03-25T19:40:54.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:40:54.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:40:54.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 10 monclient: shutdown 2024-03-25T19:40:54.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:54.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 >> v1:172.21.15.53:6789/0 conn(0x7f38f80b4700 legacy=0x7f38f8136060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 shutdown_connections 2024-03-25T19:40:54.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 >> 172.21.15.53:0/280891295 conn(0x7f38f80668e0 msgr2=0x7f38f813c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:54.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 shutdown_connections 2024-03-25T19:40:54.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 1 -- 172.21.15.53:0/280891295 wait complete. 2024-03-25T19:40:54.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.345+0000 7f38fcc6a640 1 librados: shutdown 2024-03-25T19:40:54.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 32f1487e-1fcc-417d-b7e3-e42a604b5451 6 2024-03-25T19:40:54.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.455+0000 7f1c421c8640 10 monclient: get_monmap_and_config 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.455+0000 7f1c421c8640 10 monclient: build_initial_monmap 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.455+0000 7f1c421c8640 10 monclient: monmap: 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:54.456189+0000 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:54.456189+0000 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:54.455 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 1 Processor -- start 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 1 -- start start 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 10 monclient: init 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 10 monclient(hunting): picked mon.noname-b con 0x7f1c3c0b4700 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 10 monclient(hunting): picked mon.noname-c con 0x7f1c3c142a20 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3c0a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0a9d60 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c13ff10 con 0x7f1c3c142a20 2024-03-25T19:40:54.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c0aa1f0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:54.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.456+0000 7f1c421c8640 10 monclient(hunting): authenticate will time out at 4888.609455s 2024-03-25T19:40:54.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.457+0000 7f1c32ffd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1c3c142a20 0x7f1c3c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42704/0 (socket says 172.21.15.53:42704) 2024-03-25T19:40:54.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.457+0000 7f1c32ffd640 1 -- 172.21.15.53:0/2226689660 learned_addr learned my addr 172.21.15.53:0/2226689660 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268493292 0 0) 0x7f1c3c13ff10 con 0x7f1c3c142a20 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c28003520 con 0x7f1c3c142a20 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178263078 0 0) 0x7f1c3c0aa1f0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:40:54.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.459+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3c13ff10 con 0x7f1c3c0b4700 2024-03-25T19:40:54.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.460+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325070549 0 0) 0x7f1c3c0a9d60 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.460+0000 7f1c3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.460+0000 7f1c3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.460+0000 7f1c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.460+0000 7f1c3affd640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:40:54.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.460+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3c0aa1f0 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 277842506 0 0) 0x7f1c3c13ff10 con 0x7f1c3c0b4700 2024-03-25T19:40:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 >> v1:172.21.15.53:6789/0 conn(0x7f1c3c142a20 legacy=0x7f1c3c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 >> v1:172.21.15.18:6789/0 conn(0x7f1c3c0a96b0 legacy=0x7f1c3c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 1 monclient: found mon.noname-b 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c0b4ae0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 10 monclient: _finish_auth 0 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1c2c0015f0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c3affd640 10 monclient: dump: 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:54.463 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c421c8640 5 monclient: authenticate success, global_id 4145 2024-03-25T19:40:54.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.463+0000 7f1c421c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c2c001750 con 0x7f1c3c0b4700 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c3affd640 1 -- 172.21.15.53:0/2226689660 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1c2c001b60 con 0x7f1c3c0b4700 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c3affd640 10 monclient: dump: 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 4 set_mon_vals no callback set 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 10 monclient: shutdown 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 1 -- 172.21.15.53:0/2226689660 >> v1:172.21.15.18:6790/0 conn(0x7f1c3c0b4700 legacy=0x7f1c3c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 1 -- 172.21.15.53:0/2226689660 shutdown_connections 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 1 -- 172.21.15.53:0/2226689660 >> 172.21.15.53:0/2226689660 conn(0x7f1c3c0668e0 msgr2=0x7f1c3c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 1 -- 172.21.15.53:0/2226689660 shutdown_connections 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.465+0000 7f1c421c8640 1 -- 172.21.15.53:0/2226689660 wait complete. 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.466+0000 7f1c421c8640 10 monclient: build_initial_monmap 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.466+0000 7f1c421c8640 10 monclient: monmap: 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.466+0000 7f1c421c8640 1 librados: starting msgr at 2024-03-25T19:40:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.466+0000 7f1c421c8640 1 librados: starting objecter 2024-03-25T19:40:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.466+0000 7f1c421c8640 1 Processor -- start 2024-03-25T19:40:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 1 -- start start 2024-03-25T19:40:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 1 librados: setting wanted keys 2024-03-25T19:40:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 1 librados: calling monclient init 2024-03-25T19:40:54.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 10 monclient: init 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3c0a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 10 monclient(hunting): picked mon.noname-c con 0x7f1c3c0b4700 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 10 monclient(hunting): picked mon.noname-b con 0x7f1c3c142a20 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c138ae0 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c139680 con 0x7f1c3c142a20 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3c1386e0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.467+0000 7f1c421c8640 10 monclient(hunting): authenticate will time out at 4888.620442s 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.468+0000 7f1c32ffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1c3c0b4700 0x7f1c3c137fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.53:49680/0 (socket says 172.21.15.53:49680) 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.468+0000 7f1c32ffd640 1 -- 172.21.15.53:0/3615099179 learned_addr learned my addr 172.21.15.53:0/3615099179 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.468+0000 7f1c38ff9640 10 client.?.objecter ms_handle_connect 0x7f1c3c0b4700 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.468+0000 7f1c38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.468+0000 7f1c38ff9640 10 client.?.objecter ms_handle_connect 0x7f1c3c0a96b0 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.468+0000 7f1c38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.469+0000 7f1c38ff9640 10 client.?.objecter ms_handle_connect 0x7f1c3c142a20 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.469+0000 7f1c38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515513748 0 0) 0x7f1c3c1386e0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c1c004eb0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852008214 0 0) 0x7f1c3c138ae0 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.470+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3c1386e0 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.471+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480175311 0 0) 0x7f1c3c139680 con 0x7f1c3c142a20 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.471+0000 7f1c38ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.471+0000 7f1c38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.471+0000 7f1c38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.471+0000 7f1c38ff9640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:40:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.471+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3c138ae0 con 0x7f1c3c142a20 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773631035 0 0) 0x7f1c1c004eb0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c3c139680 con 0x7f1c3c0b4700 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1537718340 0 0) 0x7f1c3c1386e0 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c1c004eb0 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1c24001990 con 0x7f1c3c0b4700 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1c2c001a60 con 0x7f1c3c0a96b0 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.475+0000 7f1c38ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:54.477 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.476+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1694330507 0 0) 0x7f1c3c138ae0 con 0x7f1c3c142a20 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.476+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c3c1386e0 con 0x7f1c3c142a20 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.476+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1c34001bd0 con 0x7f1c3c142a20 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.476+0000 7f1c38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.476+0000 7f1c38ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.476+0000 7f1c38ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.478 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:54.479 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:54.479 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:54.479 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:54.479 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3370038100 0 0) 0x7f1c3c139680 con 0x7f1c3c0b4700 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 >> v1:172.21.15.53:6789/0 conn(0x7f1c3c142a20 legacy=0x7f1c3c13be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 >> v1:172.21.15.18:6789/0 conn(0x7f1c3c0a96b0 legacy=0x7f1c3c138f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 1 monclient: found mon.c 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c13cbc0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 10 monclient: _finish_auth 0 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:54.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 5 monclient: authenticate success, global_id 4148 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 10 monclient: _renew_subs 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3c1356f0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 10 monclient: _renew_subs 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3c13c6f0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.479+0000 7f1c421c8640 1 librados: init done 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.480+0000 7f1c421c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.480+0000 7f1c421c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.480+0000 7f1c421c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:54.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.480+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3c04f730 con 0x7f1c3c0b4700 2024-03-25T19:40:54.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c240047e0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:54.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1c24004af0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:54.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 10 monclient: dump: 2024-03-25T19:40:54.502 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c3a7fc640 4 set_mon_vals no callback set 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1c240050a0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3178198129 0 0) 0x7f1c240067f0 con 0x7f1c3c0b4700 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 10 client.4148.objecter ms_dispatch 0x7f1c3c000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 3 client.4148.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:40:54.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 3 client.4148.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:40:54.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.502+0000 7f1c38ff9640 20 client.4148.objecter dump_active .. 0 homeless 2024-03-25T19:40:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.682+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1c24006310 con 0x7f1c3c0b4700 2024-03-25T19:40:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.682+0000 7f1c38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.682+0000 7f1c38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.757+0000 7f1c421c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "32f1487e-1fcc-417d-b7e3-e42a604b5451", "id": 6}] 2024-03-25T19:40:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.757+0000 7f1c421c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "32f1487e-1fcc-417d-b7e3-e42a604b5451", "id": 6}] 2024-03-25T19:40:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.757+0000 7f1c421c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.757+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd new", "uuid": "32f1487e-1fcc-417d-b7e3-e42a604b5451", "id": 6} v 0) v1 -- 0x7f1c3c135a90 con 0x7f1c3c0b4700 2024-03-25T19:40:54.828 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:54.829+0000 7fc741237040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:40:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.907+0000 7f1c38ff9640 1 -- 172.21.15.53:0/3615099179 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "32f1487e-1fcc-417d-b7e3-e42a604b5451", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3185292764 0 1113811849) 0x7f1c24006a00 con 0x7f1c3c0b4700 2024-03-25T19:40:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.907+0000 7f1c38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "32f1487e-1fcc-417d-b7e3-e42a604b5451", "id": 6}] 2024-03-25T19:40:54.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.907+0000 7f1c38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:54.907 INFO:teuthology.orchestra.run.smithi053.stdout:6 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 10 librados: watch_flush enter 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 10 librados: watch_flush exit 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 20 client.4148.objecter shutdown clearing up homeless session... 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 10 client.4148.objecter successfully canceled tick 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 10 monclient: shutdown 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 >> v1:172.21.15.18:6790/0 conn(0x7f1c3c0b4700 legacy=0x7f1c3c137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 shutdown_connections 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 >> 172.21.15.53:0/3615099179 conn(0x7f1c3c0668e0 msgr2=0x7f1c3c0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 shutdown_connections 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 1 -- 172.21.15.53:0/3615099179 wait complete. 2024-03-25T19:40:54.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:54.908+0000 7f1c421c8640 1 librados: shutdown 2024-03-25T19:40:54.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new c0d8be8d-8904-4537-9d20-c44e153b010b 7 2024-03-25T19:40:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.019+0000 7fb39eb23640 10 monclient: get_monmap_and_config 2024-03-25T19:40:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.019+0000 7fb39eb23640 10 monclient: build_initial_monmap 2024-03-25T19:40:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.019+0000 7fb39eb23640 10 monclient: monmap: 2024-03-25T19:40:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:55.020159+0000 2024-03-25T19:40:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:55.020159+0000 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.019+0000 7fb39eb23640 1 Processor -- start 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 1 -- start start 2024-03-25T19:40:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 10 monclient: init 2024-03-25T19:40:55.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:55.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:55.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 10 monclient(hunting): picked mon.noname-c con 0x7fb3980b4700 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 10 monclient(hunting): picked mon.noname-a con 0x7fb398142a20 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 10 monclient(hunting): picked mon.noname-b con 0x7fb3980a96b0 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3980a9d60 con 0x7fb398142a20 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39813ff10 con 0x7fb3980b4700 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3980aa1f0 con 0x7fb3980a96b0 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.020+0000 7fb39eb23640 10 monclient(hunting): authenticate will time out at 4889.173464s 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.021+0000 7fb39c8bf640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb3980b4700 0x7fb398142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42736/0 (socket says 172.21.15.53:42736) 2024-03-25T19:40:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.021+0000 7fb39c8bf640 1 -- 172.21.15.53:0/2693872171 learned_addr learned my addr 172.21.15.53:0/2693872171 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.023+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262129566 0 0) 0x7fb3980a9d60 con 0x7fb398142a20 2024-03-25T19:40:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.023+0000 7fb3977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.023+0000 7fb3977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.023+0000 7fb3977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.023+0000 7fb3977fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.023+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb388003520 con 0x7fb398142a20 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579253674 0 0) 0x7fb39813ff10 con 0x7fb3980b4700 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3980a9d60 con 0x7fb3980b4700 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102139493 0 0) 0x7fb3980aa1f0 con 0x7fb3980a96b0 2024-03-25T19:40:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.024+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb39813ff10 con 0x7fb3980a96b0 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1109822173 0 0) 0x7fb3980a9d60 con 0x7fb3980b4700 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 >> v1:172.21.15.18:6790/0 conn(0x7fb3980a96b0 legacy=0x7fb39813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 >> v1:172.21.15.18:6789/0 conn(0x7fb398142a20 legacy=0x7fb39813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3980b4ae0 con 0x7fb3980b4700 2024-03-25T19:40:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb38c001670 con 0x7fb3980b4700 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb39eb23640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb39eb23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb3977fe640 10 monclient: dump: 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.031 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:55.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.027+0000 7fb39eb23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb38c0017d0 con 0x7fb3980b4700 2024-03-25T19:40:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb3977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb3977fe640 1 -- 172.21.15.53:0/2693872171 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb38c001be0 con 0x7fb3980b4700 2024-03-25T19:40:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb3977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb3977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb3977fe640 10 monclient: dump: 2024-03-25T19:40:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:55.034 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb39eb23640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb39eb23640 4 set_mon_vals no callback set 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb39eb23640 10 monclient: shutdown 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb39eb23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:55.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.029+0000 7fb39eb23640 1 -- 172.21.15.53:0/2693872171 >> v1:172.21.15.53:6789/0 conn(0x7fb3980b4700 legacy=0x7fb398142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.030+0000 7fb39eb23640 1 -- 172.21.15.53:0/2693872171 shutdown_connections 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.030+0000 7fb39eb23640 1 -- 172.21.15.53:0/2693872171 >> 172.21.15.53:0/2693872171 conn(0x7fb3980668e0 msgr2=0x7fb398066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.030+0000 7fb39eb23640 1 -- 172.21.15.53:0/2693872171 shutdown_connections 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.030+0000 7fb39eb23640 1 -- 172.21.15.53:0/2693872171 wait complete. 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.030+0000 7fb39eb23640 10 monclient: build_initial_monmap 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.030+0000 7fb39eb23640 10 monclient: monmap: 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 0 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 0.000000 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:created 0.000000 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 1 2024-03-25T19:40:55.036 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.030+0000 7fb39eb23640 1 librados: starting msgr at 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.030+0000 7fb39eb23640 1 librados: starting objecter 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 1 Processor -- start 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 1 -- start start 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 1 librados: setting wanted keys 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 1 librados: calling monclient init 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 10 monclient: init 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:55.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 10 monclient(hunting): picked mon.noname-b con 0x7fb3980a96b0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 10 monclient(hunting): picked mon.noname-c con 0x7fb3980b4700 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 10 monclient(hunting): picked mon.noname-a con 0x7fb398142a20 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb398135f10 con 0x7fb398142a20 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb398139d70 con 0x7fb3980a96b0 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39813a050 con 0x7fb3980b4700 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.031+0000 7fb39eb23640 10 monclient(hunting): authenticate will time out at 4889.184601s 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.032+0000 7fb397fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fb3980b4700 0x7fb3981db320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.53:49700/0 (socket says 172.21.15.53:49700) 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.032+0000 7fb397fff640 1 -- 172.21.15.53:0/449850591 learned_addr learned my addr 172.21.15.53:0/449850591 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.032+0000 7fb3957fa640 10 client.?.objecter ms_handle_connect 0x7fb3980b4700 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.032+0000 7fb3957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.032+0000 7fb3957fa640 10 client.?.objecter ms_handle_connect 0x7fb398142a20 2024-03-25T19:40:55.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.032+0000 7fb3957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.039 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:55.033+0000 7f123ec3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:40:55.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.033+0000 7fb3957fa640 10 client.?.objecter ms_handle_connect 0x7fb3980a96b0 2024-03-25T19:40:55.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.033+0000 7fb3957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886349474 0 0) 0x7fb39813a050 con 0x7fb3980b4700 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb37c004eb0 con 0x7fb3980b4700 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982531694 0 0) 0x7fb398135f10 con 0x7fb398142a20 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.034+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb39813a050 con 0x7fb398142a20 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.035+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606747389 0 0) 0x7fb398139d70 con 0x7fb3980a96b0 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.035+0000 7fb3957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.035+0000 7fb3957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.035+0000 7fb3957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.035+0000 7fb3957fa640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.035+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb398135f10 con 0x7fb3980a96b0 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444209342 0 0) 0x7fb37c004eb0 con 0x7fb3980b4700 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb398139d70 con 0x7fb3980b4700 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb384001990 con 0x7fb3980b4700 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:55.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 10 monclient(hunting): dump: 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:55.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202418769 0 0) 0x7fb39813a050 con 0x7fb398142a20 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb37c004eb0 con 0x7fb398142a20 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb390001b50 con 0x7fb398142a20 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.039+0000 7fb3957fa640 10 monclient(hunting): dump: 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.045 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.040+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4056720845 0 0) 0x7fb398135f10 con 0x7fb3980a96b0 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.040+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb39813a050 con 0x7fb3980a96b0 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.040+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb38c001ae0 con 0x7fb3980a96b0 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.040+0000 7fb3957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.040+0000 7fb3957fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.040+0000 7fb3957fa640 10 monclient(hunting): dump: 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.047 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2520007239 0 0) 0x7fb37c004eb0 con 0x7fb398142a20 2024-03-25T19:40:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 >> v1:172.21.15.18:6790/0 conn(0x7fb3980b4700 legacy=0x7fb3981db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 >> v1:172.21.15.53:6789/0 conn(0x7fb3980a96b0 legacy=0x7fb398130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 1 monclient: found mon.a 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3981363f0 con 0x7fb398142a20 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb3957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 5 monclient: authenticate success, global_id 4161 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 10 monclient: _renew_subs 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb398138d80 con 0x7fb398142a20 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 10 monclient: _renew_subs 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb398138670 con 0x7fb398142a20 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.043+0000 7fb39eb23640 1 librados: init done 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.044+0000 7fb39eb23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.044+0000 7fb39eb23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.044+0000 7fb39eb23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.044+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb398141220 con 0x7fb398142a20 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb390001fa0 con 0x7fb398142a20 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb390003b10 con 0x7fb398142a20 2024-03-25T19:40:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 10 monclient: dump: 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:epoch 1 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:election_strategy: 3 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:40:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3967fc640 4 set_mon_vals no callback set 2024-03-25T19:40:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb390004000 con 0x7fb398142a20 2024-03-25T19:40:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 51942739 0 0) 0x7fb390005780 con 0x7fb398142a20 2024-03-25T19:40:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 10 client.4161.objecter ms_dispatch 0x7fb398000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:40:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:40:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:40:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.047+0000 7fb3957fa640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-25T19:40:55.120 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:55.121+0000 7fc741237040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:40:55.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.148+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb390005230 con 0x7fb398142a20 2024-03-25T19:40:55.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.148+0000 7fb3957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.148+0000 7fb3957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:55.174 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:55.175+0000 7fc741237040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:40:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.222+0000 7fb39eb23640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c0d8be8d-8904-4537-9d20-c44e153b010b", "id": 7}] 2024-03-25T19:40:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.222+0000 7fb39eb23640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c0d8be8d-8904-4537-9d20-c44e153b010b", "id": 7}] 2024-03-25T19:40:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.222+0000 7fb39eb23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.222+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c0d8be8d-8904-4537-9d20-c44e153b010b", "id": 7} v 0) v1 -- 0x7fb3980b7420 con 0x7fb398142a20 2024-03-25T19:40:55.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.271+0000 7fb3957fa640 1 -- 172.21.15.53:0/449850591 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c0d8be8d-8904-4537-9d20-c44e153b010b", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2503889515 0 1371665406) 0x7fb3900054f0 con 0x7fb398142a20 2024-03-25T19:40:55.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.271+0000 7fb3957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c0d8be8d-8904-4537-9d20-c44e153b010b", "id": 7}] 2024-03-25T19:40:55.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.271+0000 7fb3957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:55.271 INFO:teuthology.orchestra.run.smithi053.stdout:7 2024-03-25T19:40:55.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 10 librados: watch_flush enter 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 10 librados: watch_flush exit 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 10 client.4161.objecter successfully canceled tick 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 10 monclient: shutdown 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 >> v1:172.21.15.18:6789/0 conn(0x7fb398142a20 legacy=0x7fb3981dba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 shutdown_connections 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 >> 172.21.15.53:0/449850591 conn(0x7fb3980668e0 msgr2=0x7fb398130c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 shutdown_connections 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 1 -- 172.21.15.53:0/449850591 wait complete. 2024-03-25T19:40:55.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-25T19:40:55.272+0000 7fb39eb23640 1 librados: shutdown 2024-03-25T19:40:55.285 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:55.285 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:40:55.285 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:40:55.288 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:40:55.288 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:55.288 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:40:55.288 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:40:55.290 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:40:55.290 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:55.290 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:40:55.291 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:40:55.292 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:40:55.292 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:55.293 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:40:55.293 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:40:55.295 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:40:55.295 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:55.295 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:40:55.295 DEBUG:teuthology.orchestra.run.smithi053:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:40:55.298 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:40:55.298 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:55.298 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:40:55.298 DEBUG:teuthology.orchestra.run.smithi053:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:40:55.299 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:55.300+0000 7fc741237040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:40:55.300 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:40:55.300 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:55.300 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:40:55.301 DEBUG:teuthology.orchestra.run.smithi053:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:40:55.302 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:40:55.303 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:40:55.303 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:40:55.303 DEBUG:teuthology.orchestra.run.smithi053:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:40:55.305 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:40:55.305 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:55.328 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:55.329+0000 7f123ec3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:40:55.395 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:55.395+0000 7f123ec3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.465+0000 7f48230d9640 10 monclient: get_monmap_and_config 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.465+0000 7f48230d9640 10 monclient: build_initial_monmap 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.465+0000 7f48230d9640 10 monclient: monmap: 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:55.466094+0000 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:55.466094+0000 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:55.465 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:55.466 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:55.466 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:55.467 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.466+0000 7f48230d9640 1 Processor -- start 2024-03-25T19:40:55.467 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.466+0000 7f48230d9640 1 -- start start 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.466+0000 7f48230d9640 10 monclient: init 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 10 monclient(hunting): picked mon.noname-a con 0x7f481c0b4700 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 10 monclient(hunting): picked mon.noname-b con 0x7f481c142a20 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 10 monclient(hunting): picked mon.noname-c con 0x7f481c0a96b0 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0a9d60 con 0x7f481c0b4700 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c13ff10 con 0x7f481c0a96b0 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0aa1f0 con 0x7f481c142a20 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:55.468 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.467+0000 7f48230d9640 10 monclient(hunting): authenticate will time out at 4882.594092s 2024-03-25T19:40:55.573 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.573+0000 7f4821676640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f481c0a96b0 0x7f481c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:52178/0 (socket says 172.21.15.18:52178) 2024-03-25T19:40:55.573 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.573+0000 7f4821676640 1 -- 172.21.15.18:0/1435709360 learned_addr learned my addr 172.21.15.18:0/1435709360 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:40:55.583 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.583+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959410499 0 0) 0x7f481c13ff10 con 0x7f481c0a96b0 2024-03-25T19:40:55.583 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.583+0000 7f481b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.583 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.583+0000 7f481b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.583 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.583+0000 7f481b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.583 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.583+0000 7f481b7fe640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:40:55.583 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.583+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480c003520 con 0x7f481c0a96b0 2024-03-25T19:40:55.589 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1437058028 0 0) 0x7f480c003520 con 0x7f481c0a96b0 2024-03-25T19:40:55.589 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 >> v1:172.21.15.18:6790/0 conn(0x7f481c142a20 legacy=0x7f481c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:55.589 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 >> v1:172.21.15.18:6789/0 conn(0x7f481c0b4700 legacy=0x7f481c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:55.589 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:55.589 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:55.589 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:55.589 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:40:55.589 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0b4ae0 con 0x7f481c0a96b0 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4814001a10 con 0x7f481c0a96b0 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f481b7fe640 10 monclient: dump: 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:55.590 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.591 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.591 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.591 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:55.591 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f48230d9640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:40:55.591 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.588+0000 7f48230d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:55.599 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.598+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4814001be0 con 0x7f481c0a96b0 2024-03-25T19:40:55.599 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.598+0000 7f481b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:55.599 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.598+0000 7f481b7fe640 1 -- 172.21.15.18:0/1435709360 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4814002030 con 0x7f481c0a96b0 2024-03-25T19:40:55.599 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.598+0000 7f481b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.598+0000 7f481b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.598+0000 7f481b7fe640 10 monclient: dump: 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.599+0000 7f48230d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:55.600 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.599+0000 7f48230d9640 4 set_mon_vals no callback set 2024-03-25T19:40:55.601 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.599+0000 7f48230d9640 10 monclient: shutdown 2024-03-25T19:40:55.601 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.599+0000 7f48230d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:55.601 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.599+0000 7f48230d9640 1 -- 172.21.15.18:0/1435709360 >> v1:172.21.15.53:6789/0 conn(0x7f481c0a96b0 legacy=0x7f481c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.601 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.600+0000 7f48230d9640 1 -- 172.21.15.18:0/1435709360 shutdown_connections 2024-03-25T19:40:55.601 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.600+0000 7f48230d9640 1 -- 172.21.15.18:0/1435709360 >> 172.21.15.18:0/1435709360 conn(0x7f481c0668e0 msgr2=0x7f481c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:55.601 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.600+0000 7f48230d9640 1 -- 172.21.15.18:0/1435709360 shutdown_connections 2024-03-25T19:40:55.601 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.600+0000 7f48230d9640 1 -- 172.21.15.18:0/1435709360 wait complete. 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 10 monclient: build_initial_monmap 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 10 monclient: monmap: 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 0.000000 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:created 0.000000 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:55.603 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 1 librados: starting msgr at 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 1 librados: starting objecter 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 1 Processor -- start 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 1 -- start start 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 1 librados: setting wanted keys 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 1 librados: calling monclient init 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.601+0000 7f48230d9640 10 monclient: init 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.602+0000 7f48230d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.602+0000 7f48230d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.603+0000 7f48230d9640 10 monclient(hunting): picked mon.noname-a con 0x7f481c0a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.603+0000 7f48230d9640 10 monclient(hunting): picked mon.noname-c con 0x7f481c0b4700 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.603+0000 7f48230d9640 10 monclient(hunting): picked mon.noname-b con 0x7f481c142a20 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.603+0000 7f48230d9640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0adea0 con 0x7f481c0a96b0 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.603+0000 7f48230d9640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0abf30 con 0x7f481c142a20 2024-03-25T19:40:55.604 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.603+0000 7f48230d9640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0ae0f0 con 0x7f481c0b4700 2024-03-25T19:40:55.605 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.603+0000 7f48230d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:55.605 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.603+0000 7f48230d9640 10 monclient(hunting): authenticate will time out at 4882.730541s 2024-03-25T19:40:55.605 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-25T19:40:55.605+0000 7f123ec3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:40:55.608 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.609+0000 7f481bfff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f481c0b4700 0x7f481c0acac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54148/0 (socket says 172.21.15.18:54148) 2024-03-25T19:40:55.608 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.609+0000 7f481bfff640 1 -- 172.21.15.18:0/2211938880 learned_addr learned my addr 172.21.15.18:0/2211938880 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:40:55.609 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.610+0000 7f48197fa640 10 client.?.objecter ms_handle_connect 0x7f481c0b4700 2024-03-25T19:40:55.609 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.610+0000 7f48197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.613 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.613+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932165796 0 0) 0x7f481c0ae0f0 con 0x7f481c0b4700 2024-03-25T19:40:55.613 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.613+0000 7f48197fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:55.614 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.613+0000 7f48197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:55.614 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.613+0000 7f48197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:55.614 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.613+0000 7f48197fa640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:40:55.614 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.614+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800004eb0 con 0x7f481c0b4700 2024-03-25T19:40:55.627 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 10 client.?.objecter ms_handle_connect 0x7f481c142a20 2024-03-25T19:40:55.627 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:55.627 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3241486876 0 0) 0x7f4800004eb0 con 0x7f481c0b4700 2024-03-25T19:40:55.627 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f481c0ae0f0 con 0x7f481c0b4700 2024-03-25T19:40:55.627 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4808001960 con 0x7f481c0b4700 2024-03-25T19:40:55.627 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 10 monclient(hunting): dump: 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:55.628 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.627+0000 7f48197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:55.634 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4008917462 0 0) 0x7f481c0ae0f0 con 0x7f481c0b4700 2024-03-25T19:40:55.634 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 >> v1:172.21.15.53:6789/0 conn(0x7f481c142a20 legacy=0x7f481c1353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 >> v1:172.21.15.18:6789/0 conn(0x7f481c0a96b0 legacy=0x7f481c13ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 1 monclient: found mon.c 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c136080 con 0x7f481c0b4700 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.633+0000 7f48197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 5 monclient: authenticate success, global_id 4157 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 10 monclient: _renew_subs 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c0ad1d0 con 0x7f481c0b4700 2024-03-25T19:40:55.635 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 10 monclient: _renew_subs 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c135c90 con 0x7f481c0b4700 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.634+0000 7f48230d9640 1 librados: init done 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.635+0000 7f48230d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.635+0000 7f48230d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.635+0000 7f48230d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.635+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c0ae610 con 0x7f481c0b4700 2024-03-25T19:40:55.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.637+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4808001c40 con 0x7f481c0b4700 2024-03-25T19:40:55.637 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.637+0000 7f48197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:55.637 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.637+0000 7f481affd640 4 set_mon_vals no callback set 2024-03-25T19:40:55.697 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.698+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f48080038a0 con 0x7f481c0b4700 2024-03-25T19:40:55.697 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.698+0000 7f48197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.698+0000 7f48197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.698+0000 7f48197fa640 10 monclient: dump: 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:55.698 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:40:55.699 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.698+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 4177718251 0 0) 0x7f4808003a80 con 0x7f481c0b4700 2024-03-25T19:40:55.701 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.701+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f4808015650 con 0x7f481c0b4700 2024-03-25T19:40:55.701 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.701+0000 7f48197fa640 10 client.4157.objecter ms_dispatch 0x7f481c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:55.701 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.701+0000 7f48197fa640 3 client.4157.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:55.701 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.701+0000 7f48197fa640 3 client.4157.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:55.701 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.701+0000 7f48197fa640 20 client.4157.objecter dump_active .. 0 homeless 2024-03-25T19:40:55.841 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.841+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4808014910 con 0x7f481c0b4700 2024-03-25T19:40:55.841 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.841+0000 7f48197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:55.841 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.841+0000 7f48197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:55.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.966+0000 7f48230d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:55.966 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.966+0000 7f48230d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:55.967 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.966+0000 7f48230d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:40:55.967 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:55.966+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f481c0abd00 con 0x7f481c0b4700 2024-03-25T19:40:56.149 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.150+0000 7f48197fa640 1 -- 172.21.15.18:0/2211938880 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f48080130f0 con 0x7f481c0b4700 2024-03-25T19:40:56.149 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.150+0000 7f48197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:56.149 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.150+0000 7f48197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:56.150 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T19:40:56.150 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 10 librados: watch_flush enter 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 10 librados: watch_flush exit 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 20 client.4157.objecter shutdown clearing up homeless session... 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 10 client.4157.objecter successfully canceled tick 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 10 monclient: shutdown 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 >> v1:172.21.15.18:6790/0 conn(0x7f481c0b4700 legacy=0x7f481c0acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 shutdown_connections 2024-03-25T19:40:56.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 >> 172.21.15.18:0/2211938880 conn(0x7f481c0668e0 msgr2=0x7f481c13f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:56.152 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 shutdown_connections 2024-03-25T19:40:56.152 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.151+0000 7f48230d9640 1 -- 172.21.15.18:0/2211938880 wait complete. 2024-03-25T19:40:56.152 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T19:40:56.152+0000 7f48230d9640 1 librados: shutdown 2024-03-25T19:40:56.167 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:40:56.167 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:40:56.973 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:56.996 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:56.997+0000 7fc6e1f42640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.125+0000 7f92a5edd640 10 monclient: get_monmap_and_config 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.125+0000 7f92a5edd640 10 monclient: build_initial_monmap 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.125+0000 7f92a5edd640 10 monclient: monmap: 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:57.126386+0000 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:57.126386+0000 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:40:57.125 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:40:57.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.126+0000 7f92a5edd640 1 Processor -- start 2024-03-25T19:40:57.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.126+0000 7f92a5edd640 1 -- start start 2024-03-25T19:40:57.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.126+0000 7f92a5edd640 10 monclient: init 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 10 monclient(hunting): picked mon.noname-b con 0x7f92a00b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 10 monclient(hunting): picked mon.noname-a con 0x7f92a00b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 10 monclient(hunting): picked mon.noname-c con 0x7f92a014be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a00af940 con 0x7f92a00b1730 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a00b0a70 con 0x7f92a014be90 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a00b0d50 con 0x7f92a00b4d60 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:57.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.127+0000 7f92a5edd640 10 monclient(hunting): authenticate will time out at 4884.254409s 2024-03-25T19:40:57.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.128+0000 7f929f7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f92a00b4d60 0x7f92a013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:34948/0 (socket says 172.21.15.18:34948) 2024-03-25T19:40:57.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.128+0000 7f929f7fe640 1 -- 172.21.15.18:0/1681993793 learned_addr learned my addr 172.21.15.18:0/1681993793 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:40:57.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.132+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212082672 0 0) 0x7f92a00af940 con 0x7f92a00b1730 2024-03-25T19:40:57.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.132+0000 7f929e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.132+0000 7f929e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.132+0000 7f929e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.132+0000 7f929e7fc640 10 monclient(hunting): my global_id is 4176 2024-03-25T19:40:57.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.132+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9288003520 con 0x7f92a00b1730 2024-03-25T19:40:57.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2633356214 0 0) 0x7f9288003520 con 0x7f92a00b1730 2024-03-25T19:40:57.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 >> v1:172.21.15.18:6790/0 conn(0x7f92a00b4d60 legacy=0x7f92a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 >> v1:172.21.15.53:6789/0 conn(0x7f92a014be90 legacy=0x7f92a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:40:57.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:57.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:40:57.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:57.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:40:57.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a00b1030 con 0x7f92a00b1730 2024-03-25T19:40:57.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:57.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:57.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:57.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f92a5edd640 5 monclient: authenticate success, global_id 4176 2024-03-25T19:40:57.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f92a5edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9294001690 con 0x7f92a00b1730 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f929e7fc640 10 monclient: dump: 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:57.138 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:57.139 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:40:57.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.137+0000 7f92a5edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:57.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.140+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9294001960 con 0x7f92a00b1730 2024-03-25T19:40:57.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.140+0000 7f929e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:57.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.140+0000 7f929e7fc640 1 -- 172.21.15.18:0/1681993793 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9294001cf0 con 0x7f92a00b1730 2024-03-25T19:40:57.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.140+0000 7f92a5edd640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:57.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.140+0000 7f92a5edd640 4 set_mon_vals no callback set 2024-03-25T19:40:57.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f92a5edd640 10 monclient: shutdown 2024-03-25T19:40:57.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f929e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f929e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:57.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f929e7fc640 10 monclient: dump: 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f92a5edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f92a5edd640 1 -- 172.21.15.18:0/1681993793 >> v1:172.21.15.18:6789/0 conn(0x7f92a00b1730 legacy=0x7f92a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f92a5edd640 1 -- 172.21.15.18:0/1681993793 shutdown_connections 2024-03-25T19:40:57.142 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f92a5edd640 1 -- 172.21.15.18:0/1681993793 >> 172.21.15.18:0/1681993793 conn(0x7f92a00617a0 msgr2=0x7f92a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:57.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f92a5edd640 1 -- 172.21.15.18:0/1681993793 shutdown_connections 2024-03-25T19:40:57.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.141+0000 7f92a5edd640 1 -- 172.21.15.18:0/1681993793 wait complete. 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.143+0000 7f92a5edd640 10 monclient: build_initial_monmap 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.143+0000 7f92a5edd640 10 monclient: monmap: 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:40:57.144 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.143+0000 7f92a5edd640 1 librados: starting msgr at 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.143+0000 7f92a5edd640 1 librados: starting objecter 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.144+0000 7f92a5edd640 1 Processor -- start 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.144+0000 7f92a5edd640 1 -- start start 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.144+0000 7f92a5edd640 1 librados: setting wanted keys 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.144+0000 7f92a5edd640 1 librados: calling monclient init 2024-03-25T19:40:57.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.144+0000 7f92a5edd640 10 monclient: init 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.145+0000 7f92a5edd640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.145+0000 7f92a5edd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.145+0000 7f92a5edd640 10 monclient(hunting): picked mon.noname-c con 0x7f92a00b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.146+0000 7f92a5edd640 10 monclient(hunting): picked mon.noname-a con 0x7f92a00b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.146+0000 7f92a5edd640 10 monclient(hunting): picked mon.noname-b con 0x7f92a014be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.146+0000 7f92a5edd640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a0143900 con 0x7f92a00b4d60 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.146+0000 7f929f7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f92a00b1730 0x7f92a0141950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:34952/0 (socket says 172.21.15.18:34952) 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.146+0000 7f929f7fe640 1 -- 172.21.15.18:0/2864223079 learned_addr learned my addr 172.21.15.18:0/2864223079 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:40:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.146+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a0142f00 con 0x7f92a014be90 2024-03-25T19:40:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.146+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a01431e0 con 0x7f92a00b1730 2024-03-25T19:40:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.146+0000 7f92a5edd640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.147+0000 7f92a5edd640 10 monclient(hunting): authenticate will time out at 4884.273767s 2024-03-25T19:40:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.147+0000 7f9287fff640 10 client.?.objecter ms_handle_connect 0x7f92a00b1730 2024-03-25T19:40:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.147+0000 7f9287fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.147+0000 7f9287fff640 10 client.?.objecter ms_handle_connect 0x7f92a00b4d60 2024-03-25T19:40:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.147+0000 7f9287fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:57.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.150+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112222142 0 0) 0x7f92a0143900 con 0x7f92a00b4d60 2024-03-25T19:40:57.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.150+0000 7f9287fff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.150+0000 7f9287fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.150+0000 7f9287fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.150+0000 7f9287fff640 10 monclient(hunting): my global_id is 4179 2024-03-25T19:40:57.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.150+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f928c004eb0 con 0x7f92a00b4d60 2024-03-25T19:40:57.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.157+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1799711351 0 0) 0x7f928c004eb0 con 0x7f92a00b4d60 2024-03-25T19:40:57.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.157+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a0143900 con 0x7f92a00b4d60 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.158+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f92940017f0 con 0x7f92a00b4d60 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.158+0000 7f9287fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.158+0000 7f9287fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.158+0000 7f9287fff640 10 monclient(hunting): dump: 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:57.158 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:57.159 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:57.159 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:40:57.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.158+0000 7f9287fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.163+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4149458240 0 0) 0x7f92a0143900 con 0x7f92a00b4d60 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.163+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 >> v1:172.21.15.18:6790/0 conn(0x7f92a00b1730 legacy=0x7f92a0141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 >> v1:172.21.15.53:6789/0 conn(0x7f92a014be90 legacy=0x7f92a01dacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 1 monclient: found mon.a 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a0133630 con 0x7f92a00b4d60 2024-03-25T19:40:57.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f9287fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 5 monclient: authenticate success, global_id 4179 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 10 monclient: _renew_subs 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a0141060 con 0x7f92a00b4d60 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 10 client.4179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 10 monclient: _renew_subs 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a01db5e0 con 0x7f92a00b4d60 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.164+0000 7f92a5edd640 1 librados: init done 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.165+0000 7f92a5edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.165+0000 7f92a5edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.165+0000 7f92a5edd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:57.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.165+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a01daa10 con 0x7f92a00b4d60 2024-03-25T19:40:57.305 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:40:57.305+0000 7fc6e1f42640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:40:57.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.619+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92940039e0 con 0x7f92a00b4d60 2024-03-25T19:40:57.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.619+0000 7f9287fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:57.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.619+0000 7f929dffb640 4 set_mon_vals no callback set 2024-03-25T19:40:57.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.620+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9294003b50 con 0x7f92a00b4d60 2024-03-25T19:40:57.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.620+0000 7f9287fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.620+0000 7f9287fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.620+0000 7f9287fff640 10 monclient: dump: 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:40:57.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.621+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3055880226 0 0) 0x7f9294019ab0 con 0x7f92a00b4d60 2024-03-25T19:40:57.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.622+0000 7f929f7fe640 10 monclient: get_auth_request con 0x7f928c06db20 auth_method 0 2024-03-25T19:40:57.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.622+0000 7f9287fff640 10 client.4179.objecter ms_handle_connect 0x7f928c06db20 2024-03-25T19:40:57.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.628+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f9294053730 con 0x7f92a00b4d60 2024-03-25T19:40:57.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.628+0000 7f9287fff640 10 client.4179.objecter ms_dispatch 0x7f92a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:40:57.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.628+0000 7f9287fff640 3 client.4179.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:40:57.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.628+0000 7f9287fff640 3 client.4179.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:40:57.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.628+0000 7f9287fff640 20 client.4179.objecter dump_active .. 0 homeless 2024-03-25T19:40:57.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.628+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f929404f0f0 con 0x7f92a00b4d60 2024-03-25T19:40:57.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.628+0000 7f9287fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:57.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.628+0000 7f9287fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:57.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.990+0000 7f92a5edd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:57.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.990+0000 7f92a5edd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:57.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.990+0000 7f92a5edd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:40:57.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:57.990+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92a00627b0 con 0x7f92a00b4d60 2024-03-25T19:40:58.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.066+0000 7f9287fff640 1 -- 172.21.15.18:0/2864223079 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f9294051170 con 0x7f92a00b4d60 2024-03-25T19:40:58.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.066+0000 7f9287fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:58.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.066+0000 7f9287fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:58.066 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:40:58.066 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:58.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 10 librados: watch_flush enter 2024-03-25T19:40:58.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 10 librados: watch_flush exit 2024-03-25T19:40:58.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 20 client.4179.objecter shutdown clearing up homeless session... 2024-03-25T19:40:58.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 10 client.4179.objecter successfully canceled tick 2024-03-25T19:40:58.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 >> v1:172.21.15.18:6800/1520000573 conn(0x7f928c06db20 legacy=0x7f928c06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 10 monclient: shutdown 2024-03-25T19:40:58.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:58.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 >> v1:172.21.15.18:6789/0 conn(0x7f92a00b4d60 legacy=0x7f92a0132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 shutdown_connections 2024-03-25T19:40:58.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.068+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 >> 172.21.15.18:0/2864223079 conn(0x7f92a00617a0 msgr2=0x7f92a013c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:58.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.069+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 shutdown_connections 2024-03-25T19:40:58.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.069+0000 7f92a5edd640 1 -- 172.21.15.18:0/2864223079 wait complete. 2024-03-25T19:40:58.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:40:58.069+0000 7f92a5edd640 1 librados: shutdown 2024-03-25T19:40:58.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:02.259 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:06.897 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.266 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:06.892 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.329 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:06.967 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.330 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:06.957 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.419 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:07.033 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.481 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:07.095 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.678 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:07.301 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.739 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:07.362 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.802 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:07.444 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.823 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:07.455 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.862 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:07.504 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.884 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:07.515 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.888 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-25T19:41:02.880+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:02.891 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-25T19:41:02.885+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:02.895 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:07.510 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.924 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:07.549 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.957 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:07.572 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:02.987 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:07.611 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.032 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-25T19:41:03.026+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:03.160 INFO:tasks.ceph.osd.0.smithi018.stderr:2024-03-25T19:41:03.154+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:03.288 INFO:tasks.ceph.osd.3.smithi018.stderr:2024-03-25T19:41:03.282+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:03.345 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:07.983 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.356 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:07.982 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.458 INFO:tasks.ceph.osd.2.smithi018.stderr:2024-03-25T19:41:03.449+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:03.496 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:08.110 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.538 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-25T19:41:03.531+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:03.562 INFO:tasks.ceph.osd.1.smithi018.stderr:2024-03-25T19:41:03.553+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:03.618 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:08.260 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.729 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:08.367 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.732 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:08.358 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.785 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:08.408 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.873 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:08.487 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.948 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:08.580 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:03.993 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:08.635 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.000 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:08.615 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.044 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:08.668 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.094 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:08.733 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.104 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:08.731 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.151 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:08.774 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.232 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:08.845 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.313 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:08.955 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.318 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:08.949 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.366 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:08.981 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.398 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:09.036 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.401 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:09.027 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.417 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:09.042 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.466 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:09.089 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.526 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:09.140 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.578 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:09.220 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.679 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:09.311 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.681 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:09.296 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.695 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:09.333 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.699 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:09.325 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.738 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:09.363 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.738 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:09.362 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.815 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:09.429 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.846 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:09.487 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.887 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:04.946 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:09.561 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.978+0000 7fb9e275d640 10 monclient: get_monmap_and_config 2024-03-25T19:41:04.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.978+0000 7fb9e275d640 10 monclient: build_initial_monmap 2024-03-25T19:41:04.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.978+0000 7fb9e275d640 10 monclient: monmap: 2024-03-25T19:41:04.978 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:04.978 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:04.979 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:41:04.980017+0000 2024-03-25T19:41:04.979 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:41:04.980017+0000 2024-03-25T19:41:04.979 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:04.979 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:04.979 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:04.979 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:41:04.979 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:41:04.979 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:04.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.978+0000 7fb9e275d640 1 Processor -- start 2024-03-25T19:41:04.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 1 -- start start 2024-03-25T19:41:04.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 10 monclient: init 2024-03-25T19:41:04.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9dc0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 10 monclient(hunting): picked mon.noname-a con 0x7fb9dc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9dc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc142520 con 0x7fb9dc13f710 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0b28a0 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0b2a30 con 0x7fb9dc1430c0 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.979+0000 7fb9e275d640 10 monclient(hunting): authenticate will time out at 4892.107391s 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.980+0000 7fb9db7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fb9dc13f710 0x7fb9dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:53144/0 (socket says 172.21.15.18:53144) 2024-03-25T19:41:04.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.980+0000 7fb9db7fe640 1 -- 172.21.15.18:0/1411593746 learned_addr learned my addr 172.21.15.18:0/1411593746 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:04.981 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:09.613 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662205022 0 0) 0x7fb9dc0b2a30 con 0x7fb9dc1430c0 2024-03-25T19:41:04.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 10 monclient(hunting): my global_id is 4196 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c8003520 con 0x7fb9dc1430c0 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286106263 0 0) 0x7fb9dc0b28a0 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 10 monclient(hunting): my global_id is 4192 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.982+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc0b2a30 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.983+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221345455 0 0) 0x7fb9dc142520 con 0x7fb9dc13f710 2024-03-25T19:41:04.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.983+0000 7fb9daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.983+0000 7fb9daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.983+0000 7fb9daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.983+0000 7fb9daffd640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:41:04.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.983+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc0b28a0 con 0x7fb9dc13f710 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1742884289 0 0) 0x7fb9dc0b2a30 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 >> v1:172.21.15.18:6790/0 conn(0x7fb9dc1430c0 legacy=0x7fb9dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 >> v1:172.21.15.18:6789/0 conn(0x7fb9dc13f710 legacy=0x7fb9dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 1 monclient: found mon.noname-c 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0b44c0 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:04.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb9cc001670 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9daffd640 10 monclient: dump: 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:04.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9e275d640 5 monclient: authenticate success, global_id 4192 2024-03-25T19:41:04.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.985+0000 7fb9e275d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:04.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.987+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9cc0017d0 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.987+0000 7fb9daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.987+0000 7fb9daffd640 1 -- 172.21.15.18:0/1411593746 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb9cc001be0 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.987+0000 7fb9e275d640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.987+0000 7fb9e275d640 4 set_mon_vals no callback set 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.987+0000 7fb9daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.987+0000 7fb9daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.987+0000 7fb9daffd640 10 monclient: dump: 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:04.990 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 10 monclient: shutdown 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 1 -- 172.21.15.18:0/1411593746 >> v1:172.21.15.53:6789/0 conn(0x7fb9dc0a98f0 legacy=0x7fb9dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 1 -- 172.21.15.18:0/1411593746 shutdown_connections 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 1 -- 172.21.15.18:0/1411593746 >> 172.21.15.18:0/1411593746 conn(0x7fb9dc0b7ea0 msgr2=0x7fb9dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 1 -- 172.21.15.18:0/1411593746 shutdown_connections 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 1 -- 172.21.15.18:0/1411593746 wait complete. 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 10 monclient: build_initial_monmap 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.988+0000 7fb9e275d640 10 monclient: monmap: 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:41:04.991 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.989+0000 7fb9e275d640 1 librados: starting msgr at 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.989+0000 7fb9e275d640 1 librados: starting objecter 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.989+0000 7fb9e275d640 1 Processor -- start 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.989+0000 7fb9e275d640 1 -- start start 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.989+0000 7fb9e275d640 1 librados: setting wanted keys 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.989+0000 7fb9e275d640 1 librados: calling monclient init 2024-03-25T19:41:04.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.989+0000 7fb9e275d640 10 monclient: init 2024-03-25T19:41:04.993 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:09.630 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:04.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9dc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 10 monclient(hunting): picked mon.noname-a con 0x7fb9dc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9dc1430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc139330 con 0x7fb9dc13f710 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0abe70 con 0x7fb9dc1430c0 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc0ac150 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9e275d640 10 monclient(hunting): authenticate will time out at 4892.117968s 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9db7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fb9dc13f710 0x7fb9dc0ac570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:53154/0 (socket says 172.21.15.18:53154) 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.990+0000 7fb9db7fe640 1 -- 172.21.15.18:0/1452449251 learned_addr learned my addr 172.21.15.18:0/1452449251 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.991+0000 7fb9d8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9dc1430c0 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.991+0000 7fb9d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:04.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.991+0000 7fb9d8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9dc13f710 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.991+0000 7fb9d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.992+0000 7fb9d8ff9640 10 client.?.objecter ms_handle_connect 0x7fb9dc0a98f0 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.992+0000 7fb9d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.992+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832288642 0 0) 0x7fb9dc0abe70 con 0x7fb9dc1430c0 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.992+0000 7fb9d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.992+0000 7fb9d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.992+0000 7fb9d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.992+0000 7fb9d8ff9640 10 monclient(hunting): my global_id is 4195 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.992+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9bc004eb0 con 0x7fb9dc1430c0 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.993+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595876257 0 0) 0x7fb9dc139330 con 0x7fb9dc13f710 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.993+0000 7fb9d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.993+0000 7fb9d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.993+0000 7fb9d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.993+0000 7fb9d8ff9640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:41:04.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.993+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc0abe70 con 0x7fb9dc13f710 2024-03-25T19:41:04.997 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:09.623 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:04.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.995+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798712382 0 0) 0x7fb9dc0ac150 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.995+0000 7fb9d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.995+0000 7fb9d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.995+0000 7fb9d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.995+0000 7fb9d8ff9640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:41:04.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.995+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc139330 con 0x7fb9dc0a98f0 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.998+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1534488305 0 0) 0x7fb9bc004eb0 con 0x7fb9dc1430c0 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.998+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9dc0ac150 con 0x7fb9dc1430c0 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.998+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb9d4001af0 con 0x7fb9dc1430c0 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.998+0000 7fb9d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.998+0000 7fb9d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.998+0000 7fb9d8ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:04.999 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:05.000 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.000 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:05.000 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:05.000 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:05.000 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:05.000 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:05.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.998+0000 7fb9d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:04.999+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159582367 0 0) 0x7fb9dc139330 con 0x7fb9dc0a98f0 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9bc004eb0 con 0x7fb9dc0a98f0 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb9cc001ae0 con 0x7fb9dc0a98f0 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:05.001 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.002 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:05.002 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:05.002 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:05.002 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:05.002 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:05.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1490705719 0 0) 0x7fb9dc0abe70 con 0x7fb9dc13f710 2024-03-25T19:41:05.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9dc139330 con 0x7fb9dc13f710 2024-03-25T19:41:05.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb9c4001820 con 0x7fb9dc13f710 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.000+0000 7fb9d8ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:05.003 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 251590302 0 0) 0x7fb9dc0ac150 con 0x7fb9dc1430c0 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 >> v1:172.21.15.18:6790/0 conn(0x7fb9dc0a98f0 legacy=0x7fb9dc1389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 >> v1:172.21.15.18:6789/0 conn(0x7fb9dc13f710 legacy=0x7fb9dc0ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 1 monclient: found mon.b 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc0acc80 con 0x7fb9dc1430c0 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:05.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 5 monclient: authenticate success, global_id 4195 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 10 monclient: _renew_subs 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc137e30 con 0x7fb9dc1430c0 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 10 monclient: _renew_subs 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc136880 con 0x7fb9dc1430c0 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.002+0000 7fb9e275d640 1 librados: init done 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.003+0000 7fb9e275d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.003+0000 7fb9e275d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.003+0000 7fb9e275d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:05.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.003+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc0627b0 con 0x7fb9dc1430c0 2024-03-25T19:41:05.007 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:09.629 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.008 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:09.632 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.104 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:09.718 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.115 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:09.757 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.209 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:09.824 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.212+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d4001fe0 con 0x7fb9dc1430c0 2024-03-25T19:41:05.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.212+0000 7fb9d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:05.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.212+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb9d4003b30 con 0x7fb9dc1430c0 2024-03-25T19:41:05.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.212+0000 7fb9d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.212+0000 7fb9d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.212+0000 7fb9d8ff9640 10 monclient: dump: 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:05.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.212+0000 7fb9da7fc640 4 set_mon_vals no callback set 2024-03-25T19:41:05.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.262+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fb9d4019b00 con 0x7fb9dc1430c0 2024-03-25T19:41:05.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.262+0000 7fb9dbfff640 10 monclient: get_auth_request con 0x7fb9bc06e0b0 auth_method 0 2024-03-25T19:41:05.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.262+0000 7fb9d8ff9640 10 client.4195.objecter ms_handle_connect 0x7fb9bc06e0b0 2024-03-25T19:41:05.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.264+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fb9d4053800 con 0x7fb9dc1430c0 2024-03-25T19:41:05.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.264+0000 7fb9d8ff9640 10 client.4195.objecter ms_dispatch 0x7fb9dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:05.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.264+0000 7fb9d8ff9640 3 client.4195.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:05.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.264+0000 7fb9d8ff9640 3 client.4195.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:05.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.264+0000 7fb9d8ff9640 20 client.4195.objecter dump_active .. 0 homeless 2024-03-25T19:41:05.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.266+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9d404f140 con 0x7fb9dc1430c0 2024-03-25T19:41:05.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.266+0000 7fb9d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:05.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.266+0000 7fb9d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:05.270 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:09.895 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.275 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:09.899 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.276 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:09.908 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.302 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:09.940 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.306 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:09.933 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.385 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:10.026 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.400 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:10.014 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.446+0000 7fb9e275d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.446+0000 7fb9e275d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.446+0000 7fb9e275d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:05.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.446+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9dc1390e0 con 0x7fb9dc1430c0 2024-03-25T19:41:05.473 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:10.088 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.493+0000 7fb9d8ff9640 1 -- 172.21.15.18:0/1452449251 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fb9d4051240 con 0x7fb9dc1430c0 2024-03-25T19:41:05.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.493+0000 7fb9d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:05.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.493+0000 7fb9d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:05.493 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:41:05.493 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:05.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 10 librados: watch_flush enter 2024-03-25T19:41:05.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 10 librados: watch_flush exit 2024-03-25T19:41:05.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 20 client.4195.objecter shutdown clearing up homeless session... 2024-03-25T19:41:05.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 10 client.4195.objecter successfully canceled tick 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 >> v1:172.21.15.18:6800/1520000573 conn(0x7fb9bc06e0b0 legacy=0x7fb9bc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 10 monclient: shutdown 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 >> v1:172.21.15.53:6789/0 conn(0x7fb9dc1430c0 legacy=0x7fb9dc1e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 shutdown_connections 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.494+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 >> 172.21.15.18:0/1452449251 conn(0x7fb9dc0b7ea0 msgr2=0x7fb9dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.495+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 shutdown_connections 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.495+0000 7fb9e275d640 1 -- 172.21.15.18:0/1452449251 wait complete. 2024-03-25T19:41:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:05.495+0000 7fb9e275d640 1 librados: shutdown 2024-03-25T19:41:05.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:05.542 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:10.166 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.550 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:10.173 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.550 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:10.182 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.618 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:10.256 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.623 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:10.250 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.655 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:10.297 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.691 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:10.305 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.749 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:10.364 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.814 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:10.438 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.818 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:10.442 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.860 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:10.492 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.920 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:10.562 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.931 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:10.569 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.936 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:10.563 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:05.989 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:10.602 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.015 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:10.631 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.082 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:10.706 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.087 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:10.710 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.156 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:10.788 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.289 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:10.904 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.350 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:10.974 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.397 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:11.039 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.435 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:11.073 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.452 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:11.084 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.452 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:11.094 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.481 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:11.108 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.491 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:11.129 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.515 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:11.129 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.538 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:11.164 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.565 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:11.188 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.571 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:11.185 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.621 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:11.244 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.815 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:11.430 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.829 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:11.454 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.885 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:11.510 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.914 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:11.529 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:06.981 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:11.612 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:07.038 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:11.670 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.776 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:16.414 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.813 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:16.439 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.826 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:16.440 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.313 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:12.391 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:17.005 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.392 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:17.034 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.478+0000 7f7c07cd3640 10 monclient: get_monmap_and_config 2024-03-25T19:41:12.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.478+0000 7f7c07cd3640 10 monclient: build_initial_monmap 2024-03-25T19:41:12.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.478+0000 7f7c07cd3640 10 monclient: monmap: 2024-03-25T19:41:12.479 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:12.479 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:12.479 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:41:12.480223+0000 2024-03-25T19:41:12.480 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:41:12.480223+0000 2024-03-25T19:41:12.480 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:12.480 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:12.480 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:12.480 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:41:12.480 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:41:12.480 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:12.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.478+0000 7f7c07cd3640 1 Processor -- start 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 1 -- start start 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 10 monclient: init 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 10 monclient(hunting): picked mon.noname-c con 0x7f7c000b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 10 monclient(hunting): picked mon.noname-a con 0x7f7c000b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0014be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000af940 con 0x7f7c000b1730 2024-03-25T19:41:12.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000b0a70 con 0x7f7c000b4d60 2024-03-25T19:41:12.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c000b0d50 con 0x7f7c0014be90 2024-03-25T19:41:12.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.479+0000 7f7c07cd3640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:12.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.480+0000 7f7c07cd3640 10 monclient(hunting): authenticate will time out at 4899.607744s 2024-03-25T19:41:12.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.480+0000 7f7c0526e640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f7c000b1730 0x7f7c000b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:54534/0 (socket says 172.21.15.18:54534) 2024-03-25T19:41:12.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.480+0000 7f7c0526e640 1 -- 172.21.15.18:0/3263192364 learned_addr learned my addr 172.21.15.18:0/3263192364 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:12.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.483+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812441063 0 0) 0x7f7c000b0a70 con 0x7f7c000b4d60 2024-03-25T19:41:12.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.483+0000 7f7c04a6d640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.483+0000 7f7c04a6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.483+0000 7f7c04a6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.483+0000 7f7c04a6d640 10 monclient(hunting): my global_id is 4198 2024-03-25T19:41:12.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.483+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec003520 con 0x7f7c000b4d60 2024-03-25T19:41:12.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.483+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719832730 0 0) 0x7f7c000af940 con 0x7f7c000b1730 2024-03-25T19:41:12.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec003840 con 0x7f7c000b1730 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96530739 0 0) 0x7f7c000b0d50 con 0x7f7c0014be90 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:41:12.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.484+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c000af940 con 0x7f7c0014be90 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484816716 0 0) 0x7f7bec003520 con 0x7f7c000b4d60 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 >> v1:172.21.15.18:6790/0 conn(0x7f7c0014be90 legacy=0x7f7c0014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 >> v1:172.21.15.18:6789/0 conn(0x7f7c000b1730 legacy=0x7f7c000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 10 monclient: _finish_hunting 0 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 1 monclient: found mon.noname-c 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000b1030 con 0x7f7c000b4d60 2024-03-25T19:41:12.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 10 monclient: _finish_auth 0 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 10 monclient: _check_auth_tickets 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c04a6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c07cd3640 5 monclient: authenticate success, global_id 4198 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.486+0000 7f7c07cd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.487+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7bfc001630 con 0x7f7c000b4d60 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.487+0000 7f7c04a6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.487+0000 7f7c04a6d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.487+0000 7f7c04a6d640 10 monclient: dump: 2024-03-25T19:41:12.489 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:12.490 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:12.490 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.490 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.490 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.490 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:12.490 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:12.490 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:12.490 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:12.491 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:12.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.487+0000 7f7c07cd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:12.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.488+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bfc0018c0 con 0x7f7c000b4d60 2024-03-25T19:41:12.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.488+0000 7f7c04a6d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:12.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.488+0000 7f7c04a6d640 1 -- 172.21.15.18:0/3263192364 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7bfc001c50 con 0x7f7c000b4d60 2024-03-25T19:41:12.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.488+0000 7f7c04a6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.488+0000 7f7c04a6d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.488+0000 7f7c04a6d640 10 monclient: dump: 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:12.495 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.488+0000 7f7c07cd3640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.488+0000 7f7c07cd3640 4 set_mon_vals no callback set 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.489+0000 7f7c07cd3640 10 monclient: shutdown 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.489+0000 7f7c07cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.489+0000 7f7c07cd3640 1 -- 172.21.15.18:0/3263192364 >> v1:172.21.15.53:6789/0 conn(0x7f7c000b4d60 legacy=0x7f7c0013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.489+0000 7f7c07cd3640 1 -- 172.21.15.18:0/3263192364 shutdown_connections 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.489+0000 7f7c07cd3640 1 -- 172.21.15.18:0/3263192364 >> 172.21.15.18:0/3263192364 conn(0x7f7c000617a0 msgr2=0x7f7c00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.489+0000 7f7c07cd3640 1 -- 172.21.15.18:0/3263192364 shutdown_connections 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.489+0000 7f7c07cd3640 1 -- 172.21.15.18:0/3263192364 wait complete. 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.490+0000 7f7c07cd3640 10 monclient: build_initial_monmap 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.490+0000 7f7c07cd3640 10 monclient: monmap: 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:41:12.496 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.490+0000 7f7c07cd3640 1 librados: starting msgr at 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.490+0000 7f7c07cd3640 1 librados: starting objecter 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 1 Processor -- start 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 1 -- start start 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 1 librados: setting wanted keys 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 1 librados: calling monclient init 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 10 monclient: init 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 10 monclient(hunting): picked mon.noname-a con 0x7f7c000b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 10 monclient(hunting): picked mon.noname-b con 0x7f7c000b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 10 monclient(hunting): picked mon.noname-c con 0x7f7c0014be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c00135c10 con 0x7f7c000b1730 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c00142720 con 0x7f7c000b4d60 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c001428b0 con 0x7f7c0014be90 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.491+0000 7f7c07cd3640 10 monclient(hunting): authenticate will time out at 4899.619580s 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.492+0000 7f7c0526e640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7c000b4d60 0x7f7c001378e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:56348/0 (socket says 172.21.15.18:56348) 2024-03-25T19:41:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.492+0000 7f7c0526e640 1 -- 172.21.15.18:0/2541321548 learned_addr learned my addr 172.21.15.18:0/2541321548 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.492+0000 7f7bf67fc640 10 client.?.objecter ms_handle_connect 0x7f7c000b4d60 2024-03-25T19:41:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.492+0000 7f7bf67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.492+0000 7f7bf67fc640 10 client.?.objecter ms_handle_connect 0x7f7c0014be90 2024-03-25T19:41:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.492+0000 7f7bf67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.493+0000 7f7bf67fc640 10 client.?.objecter ms_handle_connect 0x7f7c000b1730 2024-03-25T19:41:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.493+0000 7f7bf67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.494+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579550897 0 0) 0x7f7c00142720 con 0x7f7c000b4d60 2024-03-25T19:41:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.494+0000 7f7bf67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.494+0000 7f7bf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.494+0000 7f7bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.494+0000 7f7bf67fc640 10 monclient(hunting): my global_id is 4201 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.494+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be4004eb0 con 0x7f7c000b4d60 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.495+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967083878 0 0) 0x7f7c001428b0 con 0x7f7c0014be90 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.495+0000 7f7bf67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.495+0000 7f7bf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.495+0000 7f7bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.495+0000 7f7bf67fc640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.495+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c00142720 con 0x7f7c0014be90 2024-03-25T19:41:12.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.496+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042851792 0 0) 0x7f7c00135c10 con 0x7f7c000b1730 2024-03-25T19:41:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.496+0000 7f7bf67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.496+0000 7f7bf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.496+0000 7f7bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.496+0000 7f7bf67fc640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:41:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.496+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c001428b0 con 0x7f7c000b1730 2024-03-25T19:41:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1213760427 0 0) 0x7f7be4004eb0 con 0x7f7c000b4d60 2024-03-25T19:41:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c00135c10 con 0x7f7c000b4d60 2024-03-25T19:41:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7bf0001970 con 0x7f7c000b4d60 2024-03-25T19:41:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 10 monclient(hunting): dump: 2024-03-25T19:41:12.505 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:12.505 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:12.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2328559770 0 0) 0x7f7c00142720 con 0x7f7c0014be90 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7be4004eb0 con 0x7f7c0014be90 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7bf8001b90 con 0x7f7c0014be90 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.500+0000 7f7bf67fc640 10 monclient(hunting): dump: 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:12.507 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:12.508 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:12.508 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:12.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 644316936 0 0) 0x7f7c00135c10 con 0x7f7c000b4d60 2024-03-25T19:41:12.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 >> v1:172.21.15.18:6790/0 conn(0x7f7c0014be90 legacy=0x7f7c00134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 >> v1:172.21.15.18:6789/0 conn(0x7f7c000b1730 legacy=0x7f7c001db510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 1 monclient: found mon.b 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00135650 con 0x7f7c000b4d60 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7bf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7c07cd3640 5 monclient: authenticate success, global_id 4201 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7c07cd3640 10 monclient: _renew_subs 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7c07cd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.503+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c00143050 con 0x7f7c000b4d60 2024-03-25T19:41:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.504+0000 7f7c07cd3640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.504+0000 7f7c07cd3640 10 monclient: _renew_subs 2024-03-25T19:41:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.504+0000 7f7c07cd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.504+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c001dae00 con 0x7f7c000b4d60 2024-03-25T19:41:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.504+0000 7f7c07cd3640 1 librados: init done 2024-03-25T19:41:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.505+0000 7f7c07cd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.505+0000 7f7c07cd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.505+0000 7f7c07cd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.505+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c00135840 con 0x7f7c000b4d60 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.520+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bf00047e0 con 0x7f7c000b4d60 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.520+0000 7f7bf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.520+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7bf0004af0 con 0x7f7c000b4d60 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.520+0000 7f7bf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.520+0000 7f7bf67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.520+0000 7f7bf67fc640 10 monclient: dump: 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:12.521 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.522 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:12.522 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:12.522 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:12.522 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:12.522 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:12.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.520+0000 7f7bf7fff640 4 set_mon_vals no callback set 2024-03-25T19:41:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.521+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f7bf001aac0 con 0x7f7c000b4d60 2024-03-25T19:41:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.522+0000 7f7c05a6f640 10 monclient: get_auth_request con 0x7f7be406df20 auth_method 0 2024-03-25T19:41:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.522+0000 7f7bf67fc640 10 client.4201.objecter ms_handle_connect 0x7f7be406df20 2024-03-25T19:41:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.523+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f7bf0054850 con 0x7f7c000b4d60 2024-03-25T19:41:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.523+0000 7f7bf67fc640 10 client.4201.objecter ms_dispatch 0x7f7c00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.523+0000 7f7bf67fc640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.523+0000 7f7bf67fc640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.523+0000 7f7bf67fc640 20 client.4201.objecter dump_active .. 0 homeless 2024-03-25T19:41:12.606 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:17.229 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.692 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:17.323 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.801+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7bf0050190 con 0x7f7c000b4d60 2024-03-25T19:41:12.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.801+0000 7f7bf67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:12.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:12.801+0000 7f7bf67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:13.111 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:17.735 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.132+0000 7f7c07cd3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.132+0000 7f7c07cd3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.132+0000 7f7c07cd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:13.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.132+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c000627b0 con 0x7f7c000b4d60 2024-03-25T19:41:13.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.158+0000 7f7bf67fc640 1 -- 172.21.15.18:0/2541321548 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f7bf0052290 con 0x7f7c000b4d60 2024-03-25T19:41:13.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.158+0000 7f7bf67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.158+0000 7f7bf67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:13.159 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:41:13.159 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:13.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.160+0000 7f7c07cd3640 10 librados: watch_flush enter 2024-03-25T19:41:13.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 10 librados: watch_flush exit 2024-03-25T19:41:13.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 20 client.4201.objecter shutdown clearing up homeless session... 2024-03-25T19:41:13.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 10 client.4201.objecter successfully canceled tick 2024-03-25T19:41:13.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 >> v1:172.21.15.18:6800/1520000573 conn(0x7f7be406df20 legacy=0x7f7be40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 10 monclient: shutdown 2024-03-25T19:41:13.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:13.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 >> v1:172.21.15.53:6789/0 conn(0x7f7c000b4d60 legacy=0x7f7c001378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 shutdown_connections 2024-03-25T19:41:13.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 >> 172.21.15.18:0/2541321548 conn(0x7f7c000617a0 msgr2=0x7f7c000b70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:13.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 shutdown_connections 2024-03-25T19:41:13.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 1 -- 172.21.15.18:0/2541321548 wait complete. 2024-03-25T19:41:13.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:13.161+0000 7f7c07cd3640 1 librados: shutdown 2024-03-25T19:41:13.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:17.386 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:22.012 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.390 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:22.017 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.528 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:22.141 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.532 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:22.145 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.569 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:22.207 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:17.573 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:22.211 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.243 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:22.884 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.247 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:22.888 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.365 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:22.980 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.369 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:22.985 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.459 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:23.082 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.462 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:23.086 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.599 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:23.231 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.605 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:23.237 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.863 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:23.487 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:18.869 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:23.493 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.978 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.083+0000 7ff82bd23640 10 monclient: get_monmap_and_config 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.083+0000 7ff82bd23640 10 monclient: build_initial_monmap 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.083+0000 7ff82bd23640 10 monclient: monmap: 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:41:20.084403+0000 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:41:20.084403+0000 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:20.083 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:20.084 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:41:20.084 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:41:20.084 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:20.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.084+0000 7ff82bd23640 1 Processor -- start 2024-03-25T19:41:20.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.084+0000 7ff82bd23640 1 -- start start 2024-03-25T19:41:20.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.084+0000 7ff82bd23640 10 monclient: init 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 10 monclient(hunting): picked mon.noname-b con 0x7ff8240b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 10 monclient(hunting): picked mon.noname-c con 0x7ff8240b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 10 monclient(hunting): picked mon.noname-a con 0x7ff82414be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8240af940 con 0x7ff82414be90 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8240b0a70 con 0x7ff8240b1730 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8240b0d50 con 0x7ff8240b4d60 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.085+0000 7ff82bd23640 10 monclient(hunting): authenticate will time out at 4907.212563s 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.086+0000 7ff82a2c0640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7ff82414be90 0x7ff82414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:37894/0 (socket says 172.21.15.18:37894) 2024-03-25T19:41:20.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.086+0000 7ff82a2c0640 1 -- 172.21.15.18:0/2598769473 learned_addr learned my addr 172.21.15.18:0/2598769473 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:20.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.088+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452960350 0 0) 0x7ff8240af940 con 0x7ff82414be90 2024-03-25T19:41:20.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.088+0000 7ff828abd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.088+0000 7ff828abd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.088+0000 7ff828abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.088+0000 7ff828abd640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:41:20.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.088+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff810003520 con 0x7ff82414be90 2024-03-25T19:41:20.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861329716 0 0) 0x7ff8240b0a70 con 0x7ff8240b1730 2024-03-25T19:41:20.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:41:20.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8240af940 con 0x7ff8240b1730 2024-03-25T19:41:20.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102268471 0 0) 0x7ff8240b0d50 con 0x7ff8240b4d60 2024-03-25T19:41:20.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:41:20.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.090+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8240b0a70 con 0x7ff8240b4d60 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1511968529 0 0) 0x7ff810003520 con 0x7ff82414be90 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 >> v1:172.21.15.18:6790/0 conn(0x7ff8240b4d60 legacy=0x7ff82413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 >> v1:172.21.15.53:6789/0 conn(0x7ff8240b1730 legacy=0x7ff8240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 1 monclient: found mon.noname-a 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8240b1030 con 0x7ff82414be90 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 10 monclient: _finish_auth 0 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:20.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff81c001a10 con 0x7ff82414be90 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff828abd640 10 monclient: dump: 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff82bd23640 5 monclient: authenticate success, global_id 4218 2024-03-25T19:41:20.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.093+0000 7ff82bd23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:20.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.095+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff81c001b70 con 0x7ff82414be90 2024-03-25T19:41:20.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.095+0000 7ff828abd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:20.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.095+0000 7ff82bd23640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:20.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.095+0000 7ff82bd23640 4 set_mon_vals no callback set 2024-03-25T19:41:20.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.096+0000 7ff828abd640 1 -- 172.21.15.18:0/2598769473 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff81c001f80 con 0x7ff82414be90 2024-03-25T19:41:20.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.096+0000 7ff828abd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.096+0000 7ff828abd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.096+0000 7ff828abd640 10 monclient: dump: 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.096+0000 7ff82bd23640 10 monclient: shutdown 2024-03-25T19:41:20.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.096+0000 7ff82bd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:20.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.096+0000 7ff82bd23640 1 -- 172.21.15.18:0/2598769473 >> v1:172.21.15.18:6789/0 conn(0x7ff82414be90 legacy=0x7ff82414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.097+0000 7ff82bd23640 1 -- 172.21.15.18:0/2598769473 shutdown_connections 2024-03-25T19:41:20.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.097+0000 7ff82bd23640 1 -- 172.21.15.18:0/2598769473 >> 172.21.15.18:0/2598769473 conn(0x7ff8240617a0 msgr2=0x7ff824061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:20.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.097+0000 7ff82bd23640 1 -- 172.21.15.18:0/2598769473 shutdown_connections 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.097+0000 7ff82bd23640 1 -- 172.21.15.18:0/2598769473 wait complete. 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.098+0000 7ff82bd23640 10 monclient: build_initial_monmap 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.099+0000 7ff82bd23640 10 monclient: monmap: 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:20.099 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.099+0000 7ff82bd23640 1 librados: starting msgr at 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.099+0000 7ff82bd23640 1 librados: starting objecter 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.099+0000 7ff82bd23640 1 Processor -- start 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.099+0000 7ff82bd23640 1 -- start start 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.099+0000 7ff82bd23640 1 librados: setting wanted keys 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.099+0000 7ff82bd23640 1 librados: calling monclient init 2024-03-25T19:41:20.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.099+0000 7ff82bd23640 10 monclient: init 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 10 monclient(hunting): picked mon.noname-b con 0x7ff8240b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 10 monclient(hunting): picked mon.noname-c con 0x7ff8240b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 10 monclient(hunting): picked mon.noname-a con 0x7ff82414be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff824140860 con 0x7ff82414be90 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff824137fd0 con 0x7ff8240b1730 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff824140380 con 0x7ff8240b4d60 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.100+0000 7ff82bd23640 10 monclient(hunting): authenticate will time out at 4907.227604s 2024-03-25T19:41:20.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.101+0000 7ff82a2c0640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7ff82414be90 0x7ff824142880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:37902/0 (socket says 172.21.15.18:37902) 2024-03-25T19:41:20.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.101+0000 7ff82a2c0640 1 -- 172.21.15.18:0/2583395292 learned_addr learned my addr 172.21.15.18:0/2583395292 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:20.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.101+0000 7ff81a7fc640 10 client.?.objecter ms_handle_connect 0x7ff82414be90 2024-03-25T19:41:20.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.102+0000 7ff81a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.102+0000 7ff81a7fc640 10 client.?.objecter ms_handle_connect 0x7ff8240b1730 2024-03-25T19:41:20.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.102+0000 7ff81a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.102+0000 7ff81a7fc640 10 client.?.objecter ms_handle_connect 0x7ff8240b4d60 2024-03-25T19:41:20.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.102+0000 7ff81a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901047463 0 0) 0x7ff824137fd0 con 0x7ff8240b1730 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff808004eb0 con 0x7ff8240b1730 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912511519 0 0) 0x7ff824140860 con 0x7ff82414be90 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:41:20.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.104+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff824137fd0 con 0x7ff82414be90 2024-03-25T19:41:20.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.105+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801438105 0 0) 0x7ff824140380 con 0x7ff8240b4d60 2024-03-25T19:41:20.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.105+0000 7ff81a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.105+0000 7ff81a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.105+0000 7ff81a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.105+0000 7ff81a7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:41:20.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.105+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff824140860 con 0x7ff8240b4d60 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.110+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2349299884 0 0) 0x7ff808004eb0 con 0x7ff8240b1730 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.110+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff824140380 con 0x7ff8240b1730 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.110+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff8200018f0 con 0x7ff8240b1730 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.110+0000 7ff81a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.110+0000 7ff81a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.110+0000 7ff81a7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.110 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:20.111 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:20.111 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:20.111 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:20.111 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:20.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.110+0000 7ff81a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.112+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2080595624 0 0) 0x7ff824140860 con 0x7ff8240b4d60 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.112+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff808004eb0 con 0x7ff8240b4d60 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.112+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1992388933 0 0) 0x7ff824137fd0 con 0x7ff82414be90 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.112+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff824140860 con 0x7ff82414be90 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.112+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff81c001e80 con 0x7ff82414be90 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.112+0000 7ff81a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.112+0000 7ff81a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.112+0000 7ff81a7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:20.112 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.113 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.113 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.113 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:20.113 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:20.113 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:20.113 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:20.113 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:20.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1102328362 0 0) 0x7ff824140380 con 0x7ff8240b1730 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 >> v1:172.21.15.18:6790/0 conn(0x7ff8240b4d60 legacy=0x7ff824142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 >> v1:172.21.15.18:6789/0 conn(0x7ff82414be90 legacy=0x7ff824142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 1 monclient: found mon.b 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff824140510 con 0x7ff8240b1730 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff81a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff82bd23640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff82bd23640 10 monclient: _renew_subs 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff82bd23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8241374a0 con 0x7ff8240b1730 2024-03-25T19:41:20.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.115+0000 7ff82bd23640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:20.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.116+0000 7ff82bd23640 10 monclient: _renew_subs 2024-03-25T19:41:20.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.116+0000 7ff82bd23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:20.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.116+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff824145000 con 0x7ff8240b1730 2024-03-25T19:41:20.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.116+0000 7ff82bd23640 1 librados: init done 2024-03-25T19:41:20.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.117+0000 7ff82bd23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:20.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.117+0000 7ff82bd23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:20.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.117+0000 7ff82bd23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:20.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.117+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8240627b0 con 0x7ff8240b1730 2024-03-25T19:41:20.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.465+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff820003610 con 0x7ff8240b1730 2024-03-25T19:41:20.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.465+0000 7ff81a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:20.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.465+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff820003920 con 0x7ff8240b1730 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.465+0000 7ff81a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.465+0000 7ff81a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.465+0000 7ff81a7fc640 10 monclient: dump: 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:20.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.465+0000 7ff81bfff640 4 set_mon_vals no callback set 2024-03-25T19:41:20.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.466+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ff8200198f0 con 0x7ff8240b1730 2024-03-25T19:41:20.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.466+0000 7ff8292be640 10 monclient: get_auth_request con 0x7ff80806e310 auth_method 0 2024-03-25T19:41:20.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.467+0000 7ff81a7fc640 10 client.4207.objecter ms_handle_connect 0x7ff80806e310 2024-03-25T19:41:20.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.473+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7ff8200535c0 con 0x7ff8240b1730 2024-03-25T19:41:20.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.473+0000 7ff81a7fc640 10 client.4207.objecter ms_dispatch 0x7ff824000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:20.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.473+0000 7ff81a7fc640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:20.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.473+0000 7ff81a7fc640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:20.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.473+0000 7ff81a7fc640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T19:41:20.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.473+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff82004ef00 con 0x7ff8240b1730 2024-03-25T19:41:20.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.473+0000 7ff81a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:20.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.473+0000 7ff81a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:20.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.712+0000 7ff82bd23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:20.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.712+0000 7ff82bd23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:20.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.712+0000 7ff82bd23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:20.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.712+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff824143340 con 0x7ff8240b1730 2024-03-25T19:41:20.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.730+0000 7ff81a7fc640 1 -- 172.21.15.18:0/2583395292 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7ff820051000 con 0x7ff8240b1730 2024-03-25T19:41:20.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.730+0000 7ff81a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:20.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.730+0000 7ff81a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:20.730 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:41:20.730 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:20.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 10 librados: watch_flush enter 2024-03-25T19:41:20.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 10 librados: watch_flush exit 2024-03-25T19:41:20.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T19:41:20.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 10 client.4207.objecter successfully canceled tick 2024-03-25T19:41:20.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 >> v1:172.21.15.18:6800/1520000573 conn(0x7ff80806e310 legacy=0x7ff8080707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 10 monclient: shutdown 2024-03-25T19:41:20.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:20.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 >> v1:172.21.15.53:6789/0 conn(0x7ff8240b1730 legacy=0x7ff824134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 shutdown_connections 2024-03-25T19:41:20.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 >> 172.21.15.18:0/2583395292 conn(0x7ff8240617a0 msgr2=0x7ff82413c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:20.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 shutdown_connections 2024-03-25T19:41:20.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 1 -- 172.21.15.18:0/2583395292 wait complete. 2024-03-25T19:41:20.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:20.732+0000 7ff82bd23640 1 librados: shutdown 2024-03-25T19:41:20.742 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:21.645 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:00:26.271 35216== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:21.800 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:00:26.413 35217== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:21.965 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:00:26.603 35215== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:22.751 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:27.393 35800== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:22.889 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:27.512 35803== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:23.056 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:00:27.671 35218== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:23.555 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:28.179 35802== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:23.696 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:28.328 35801== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:27.548 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:27.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.680+0000 7f5abc2a8640 10 monclient: get_monmap_and_config 2024-03-25T19:41:27.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.680+0000 7f5abc2a8640 10 monclient: build_initial_monmap 2024-03-25T19:41:27.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.680+0000 7f5abc2a8640 10 monclient: monmap: 2024-03-25T19:41:27.680 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:27.680 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:27.681 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:41:27.681758+0000 2024-03-25T19:41:27.681 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:41:27.681758+0000 2024-03-25T19:41:27.681 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:27.681 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:27.681 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:27.681 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:41:27.681 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:41:27.681 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.681+0000 7f5abc2a8640 1 Processor -- start 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.681+0000 7f5abc2a8640 1 -- start start 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.681+0000 7f5abc2a8640 10 monclient: init 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab40a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab41430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:27.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab4142520 con 0x7f5ab413f710 2024-03-25T19:41:27.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40b28a0 con 0x7f5ab41430c0 2024-03-25T19:41:27.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40b2a30 con 0x7f5ab40a98f0 2024-03-25T19:41:27.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.682+0000 7f5abc2a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:27.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.683+0000 7f5abc2a8640 10 monclient(hunting): authenticate will time out at 4914.809739s 2024-03-25T19:41:27.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.683+0000 7f5aba845640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5ab41430c0 0x7f5ab414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:37872/0 (socket says 172.21.15.18:37872) 2024-03-25T19:41:27.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.683+0000 7f5aba845640 1 -- 172.21.15.18:0/1650790393 learned_addr learned my addr 172.21.15.18:0/1650790393 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:27.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.766+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365769625 0 0) 0x7f5ab4142520 con 0x7f5ab413f710 2024-03-25T19:41:27.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.766+0000 7f5ab9042640 10 monclient(hunting): _negotiate 2024-03-25T19:41:27.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.766+0000 7f5ab9042640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:27.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.766+0000 7f5ab9042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:27.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.766+0000 7f5ab9042640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:41:27.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.766+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa0003520 con 0x7f5ab413f710 2024-03-25T19:41:27.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384986149 0 0) 0x7f5aa0003520 con 0x7f5ab413f710 2024-03-25T19:41:27.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 >> v1:172.21.15.18:6790/0 conn(0x7f5ab40a98f0 legacy=0x7f5ab40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 >> v1:172.21.15.53:6789/0 conn(0x7f5ab41430c0 legacy=0x7f5ab414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 10 monclient: _finish_hunting 0 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 1 monclient: found mon.noname-a 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab40b44c0 con 0x7f5ab413f710 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 10 monclient: _finish_auth 0 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 10 monclient: _check_auth_tickets 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5aa4001670 con 0x7f5ab413f710 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5ab9042640 10 monclient: dump: 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:27.772 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5abc2a8640 5 monclient: authenticate success, global_id 4224 2024-03-25T19:41:27.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.771+0000 7f5abc2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.774+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aa40017d0 con 0x7f5ab413f710 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.774+0000 7f5ab9042640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.774+0000 7f5abc2a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.774+0000 7f5abc2a8640 4 set_mon_vals no callback set 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5ab9042640 1 -- 172.21.15.18:0/1650790393 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5aa4001c20 con 0x7f5ab413f710 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5ab9042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5ab9042640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5ab9042640 10 monclient: dump: 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:27.775 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5abc2a8640 10 monclient: shutdown 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5abc2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5abc2a8640 1 -- 172.21.15.18:0/1650790393 >> v1:172.21.15.18:6789/0 conn(0x7f5ab413f710 legacy=0x7f5ab41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5abc2a8640 1 -- 172.21.15.18:0/1650790393 shutdown_connections 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.775+0000 7f5abc2a8640 1 -- 172.21.15.18:0/1650790393 >> 172.21.15.18:0/1650790393 conn(0x7f5ab40b7ea0 msgr2=0x7f5ab40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:27.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.776+0000 7f5abc2a8640 1 -- 172.21.15.18:0/1650790393 shutdown_connections 2024-03-25T19:41:27.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.776+0000 7f5abc2a8640 1 -- 172.21.15.18:0/1650790393 wait complete. 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 10 monclient: build_initial_monmap 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 10 monclient: monmap: 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 1 librados: starting msgr at 2024-03-25T19:41:27.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 1 librados: starting objecter 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 1 Processor -- start 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 1 -- start start 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 1 librados: setting wanted keys 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 1 librados: calling monclient init 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.777+0000 7f5abc2a8640 10 monclient: init 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.778+0000 7f5abc2a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.778+0000 7f5abc2a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.778+0000 7f5abc2a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab40a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.778+0000 7f5abc2a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab413f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.778+0000 7f5abc2a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab41430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.778+0000 7f5abc2a8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab41355c0 con 0x7f5ab40a98f0 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.779+0000 7f5abc2a8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab4137160 con 0x7f5ab41430c0 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.779+0000 7f5abc2a8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab40b68b0 con 0x7f5ab413f710 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.779+0000 7f5abc2a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:27.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.779+0000 7f5abc2a8640 10 monclient(hunting): authenticate will time out at 4914.906029s 2024-03-25T19:41:27.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.779+0000 7f5aba044640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f5ab40a98f0 0x7f5ab40b5a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60446/0 (socket says 172.21.15.18:60446) 2024-03-25T19:41:27.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.779+0000 7f5aba044640 1 -- 172.21.15.18:0/479269608 learned_addr learned my addr 172.21.15.18:0/479269608 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:27.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.779+0000 7f5aaaffd640 10 client.?.objecter ms_handle_connect 0x7f5ab40a98f0 2024-03-25T19:41:27.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.779+0000 7f5aaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:27.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.783+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756079936 0 0) 0x7f5ab41355c0 con 0x7f5ab40a98f0 2024-03-25T19:41:27.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.783+0000 7f5aaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:27.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.783+0000 7f5aaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:27.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.783+0000 7f5aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:27.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.783+0000 7f5aaaffd640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:41:27.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.783+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a98004eb0 con 0x7f5ab40a98f0 2024-03-25T19:41:27.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.790+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103732854 0 0) 0x7f5a98004eb0 con 0x7f5ab40a98f0 2024-03-25T19:41:27.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.790+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab41355c0 con 0x7f5ab40a98f0 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.790+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5ab0001b30 con 0x7f5ab40a98f0 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.790+0000 7f5aaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.790+0000 7f5aaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.790+0000 7f5aaaffd640 10 monclient(hunting): dump: 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:27.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.790+0000 7f5aaaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:27.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1849406771 0 0) 0x7f5ab41355c0 con 0x7f5ab40a98f0 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 >> v1:172.21.15.18:6790/0 conn(0x7f5ab413f710 legacy=0x7f5ab40b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 >> v1:172.21.15.53:6789/0 conn(0x7f5ab41430c0 legacy=0x7f5ab412fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 1 monclient: found mon.a 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab41373d0 con 0x7f5ab40a98f0 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5aaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 5 monclient: authenticate success, global_id 4227 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 10 monclient: _renew_subs 2024-03-25T19:41:27.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab41351c0 con 0x7f5ab40a98f0 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 10 monclient: _renew_subs 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab40b7980 con 0x7f5ab40a98f0 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.796+0000 7f5abc2a8640 1 librados: init done 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.797+0000 7f5abc2a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.797+0000 7f5abc2a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.797+0000 7f5abc2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:41:27.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:27.797+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab40b7c80 con 0x7f5ab40a98f0 2024-03-25T19:41:28.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.015+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ab0001c90 con 0x7f5ab40a98f0 2024-03-25T19:41:28.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.015+0000 7f5aaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.015+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5ab0003ae0 con 0x7f5ab40a98f0 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.015+0000 7f5aaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.015+0000 7f5aaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.015+0000 7f5aaaffd640 10 monclient: dump: 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:28.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.015+0000 7f5ab8841640 4 set_mon_vals no callback set 2024-03-25T19:41:28.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.100+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f5ab0003c40 con 0x7f5ab40a98f0 2024-03-25T19:41:28.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.101+0000 7f5ab9843640 10 monclient: get_auth_request con 0x7f5a9806db30 auth_method 0 2024-03-25T19:41:28.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.101+0000 7f5aaaffd640 10 client.4227.objecter ms_handle_connect 0x7f5a9806db30 2024-03-25T19:41:28.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.130+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f5ab0054990 con 0x7f5ab40a98f0 2024-03-25T19:41:28.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.130+0000 7f5aaaffd640 10 client.4227.objecter ms_dispatch 0x7f5ab4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:28.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.130+0000 7f5aaaffd640 3 client.4227.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:28.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.130+0000 7f5aaaffd640 3 client.4227.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:28.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.130+0000 7f5aaaffd640 20 client.4227.objecter dump_active .. 0 homeless 2024-03-25T19:41:28.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.130+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ab00502d0 con 0x7f5ab40a98f0 2024-03-25T19:41:28.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.130+0000 7f5aaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:28.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.130+0000 7f5aaaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:28.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.517+0000 7f5abc2a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.517+0000 7f5abc2a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.517+0000 7f5abc2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:41:28.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.517+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ab40627b0 con 0x7f5ab40a98f0 2024-03-25T19:41:28.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.543+0000 7f5aaaffd640 1 -- 172.21.15.18:0/479269608 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f5ab00523d0 con 0x7f5ab40a98f0 2024-03-25T19:41:28.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.543+0000 7f5aaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.543+0000 7f5aaaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:28.543 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:41:28.543 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:28.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 10 librados: watch_flush enter 2024-03-25T19:41:28.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 10 librados: watch_flush exit 2024-03-25T19:41:28.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 20 client.4227.objecter shutdown clearing up homeless session... 2024-03-25T19:41:28.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 10 client.4227.objecter successfully canceled tick 2024-03-25T19:41:28.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 >> v1:172.21.15.18:6800/1520000573 conn(0x7f5a9806db30 legacy=0x7f5a9806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 10 monclient: shutdown 2024-03-25T19:41:28.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:28.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 >> v1:172.21.15.18:6789/0 conn(0x7f5ab40a98f0 legacy=0x7f5ab40b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 shutdown_connections 2024-03-25T19:41:28.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 >> 172.21.15.18:0/479269608 conn(0x7f5ab40b7ea0 msgr2=0x7f5ab4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:28.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 shutdown_connections 2024-03-25T19:41:28.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 1 -- 172.21.15.18:0/479269608 wait complete. 2024-03-25T19:41:28.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:28.545+0000 7f5abc2a8640 1 librados: shutdown 2024-03-25T19:41:28.555 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:30.279 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-25T19:41:30.276+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:41:30.351 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-25T19:41:30.342+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:41:30.656 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-25T19:41:30.653+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:41:31.717 INFO:tasks.ceph.osd.0.smithi018.stderr:2024-03-25T19:41:31.714+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:41:31.788 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-25T19:41:31.786+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:41:32.133 INFO:tasks.ceph.osd.3.smithi018.stderr:2024-03-25T19:41:32.100+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:41:32.779 INFO:tasks.ceph.osd.1.smithi018.stderr:2024-03-25T19:41:32.777+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:41:33.036 INFO:tasks.ceph.osd.2.smithi018.stderr:2024-03-25T19:41:33.034+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:41:35.361 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.478+0000 7f2cdc4d3640 10 monclient: get_monmap_and_config 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.478+0000 7f2cdc4d3640 10 monclient: build_initial_monmap 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.478+0000 7f2cdc4d3640 10 monclient: monmap: 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:41:35.478905+0000 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:41:35.478905+0000 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:35.478 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:35.479 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:35.479 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:41:35.479 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:41:35.479 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:35.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.478+0000 7f2cdc4d3640 1 Processor -- start 2024-03-25T19:41:35.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.478+0000 7f2cdc4d3640 1 -- start start 2024-03-25T19:41:35.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.478+0000 7f2cdc4d3640 10 monclient: init 2024-03-25T19:41:35.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.479+0000 7f2cdc4d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.479+0000 7f2cdc4d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.480+0000 7f2cdc4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd40a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.480+0000 7f2cdc4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.480+0000 7f2cdc4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.480+0000 7f2cdc4d3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd4142520 con 0x7f2cd413f710 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.480+0000 7f2cdc4d3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd40b28a0 con 0x7f2cd40a98f0 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.480+0000 7f2cdc4d3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd40b2a30 con 0x7f2cd41430c0 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.480+0000 7f2cdc4d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:35.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.480+0000 7f2cdc4d3640 10 monclient(hunting): authenticate will time out at 4922.606864s 2024-03-25T19:41:35.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.491+0000 7f2cda26f640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2cd40a98f0 0x7f2cd40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:38870/0 (socket says 172.21.15.18:38870) 2024-03-25T19:41:35.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.491+0000 7f2cda26f640 1 -- 172.21.15.18:0/1037571915 learned_addr learned my addr 172.21.15.18:0/1037571915 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:35.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.497+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252040105 0 0) 0x7f2cd40b28a0 con 0x7f2cd40a98f0 2024-03-25T19:41:35.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.497+0000 7f2cd926d640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.497+0000 7f2cd926d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.497+0000 7f2cd926d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.497+0000 7f2cd926d640 10 monclient(hunting): my global_id is 4234 2024-03-25T19:41:35.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.497+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc0003520 con 0x7f2cd40a98f0 2024-03-25T19:41:35.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.506+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790454901 0 0) 0x7f2cd4142520 con 0x7f2cd413f710 2024-03-25T19:41:35.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.506+0000 7f2cd926d640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.506+0000 7f2cd926d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.506+0000 7f2cd926d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.506+0000 7f2cd926d640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:41:35.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.506+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd40b28a0 con 0x7f2cd413f710 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3943483835 0 0) 0x7f2cc0003520 con 0x7f2cd40a98f0 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 >> v1:172.21.15.18:6790/0 conn(0x7f2cd41430c0 legacy=0x7f2cd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 >> v1:172.21.15.18:6789/0 conn(0x7f2cd413f710 legacy=0x7f2cd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 10 monclient: _finish_hunting 0 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 1 monclient: found mon.noname-c 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd40b44c0 con 0x7f2cd40a98f0 2024-03-25T19:41:35.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 10 monclient: _finish_auth 0 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 10 monclient: _check_auth_tickets 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2cd0001690 con 0x7f2cd40a98f0 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cd926d640 10 monclient: dump: 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:35.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cdc4d3640 5 monclient: authenticate success, global_id 4234 2024-03-25T19:41:35.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.510+0000 7f2cdc4d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.516+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cd00017f0 con 0x7f2cd40a98f0 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.516+0000 7f2cd926d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.516+0000 7f2cd926d640 1 -- 172.21.15.18:0/1037571915 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2cd0001c00 con 0x7f2cd40a98f0 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.516+0000 7f2cdc4d3640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.516+0000 7f2cdc4d3640 4 set_mon_vals no callback set 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.516+0000 7f2cd926d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.516+0000 7f2cd926d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.516+0000 7f2cd926d640 10 monclient: dump: 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.516 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.517 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:35.517 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:35.517 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:35.517 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:35.517 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:35.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.517+0000 7f2cdc4d3640 10 monclient: shutdown 2024-03-25T19:41:35.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.517+0000 7f2cdc4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:35.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.517+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/1037571915 >> v1:172.21.15.53:6789/0 conn(0x7f2cd40a98f0 legacy=0x7f2cd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.517+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/1037571915 shutdown_connections 2024-03-25T19:41:35.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.517+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/1037571915 >> 172.21.15.18:0/1037571915 conn(0x7f2cd40b7ea0 msgr2=0x7f2cd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.517+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/1037571915 shutdown_connections 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.517+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/1037571915 wait complete. 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.518+0000 7f2cdc4d3640 10 monclient: build_initial_monmap 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.518+0000 7f2cdc4d3640 10 monclient: monmap: 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:41:35.519 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:35.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 librados: starting msgr at 2024-03-25T19:41:35.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 librados: starting objecter 2024-03-25T19:41:35.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 Processor -- start 2024-03-25T19:41:35.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 -- start start 2024-03-25T19:41:35.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 librados: setting wanted keys 2024-03-25T19:41:35.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 librados: calling monclient init 2024-03-25T19:41:35.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 10 monclient: init 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd40a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd41d7380 con 0x7f2cd40a98f0 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd4139840 con 0x7f2cd413f710 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd41d8200 con 0x7f2cd41430c0 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:35.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.519+0000 7f2cdc4d3640 10 monclient(hunting): authenticate will time out at 4922.646627s 2024-03-25T19:41:35.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.523+0000 7f2cdaa70640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f2cd41430c0 0x7f2cd41daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40152/0 (socket says 172.21.15.18:40152) 2024-03-25T19:41:35.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.523+0000 7f2cdaa70640 1 -- 172.21.15.18:0/2815830899 learned_addr learned my addr 172.21.15.18:0/2815830899 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:35.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.525+0000 7f2ccaffd640 10 client.?.objecter ms_handle_connect 0x7f2cd41430c0 2024-03-25T19:41:35.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.525+0000 7f2ccaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:35.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.534+0000 7f2ccaffd640 10 client.?.objecter ms_handle_connect 0x7f2cd413f710 2024-03-25T19:41:35.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.534+0000 7f2ccaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:35.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.536+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617743667 0 0) 0x7f2cd41d8200 con 0x7f2cd41430c0 2024-03-25T19:41:35.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.536+0000 7f2ccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.536+0000 7f2ccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.536+0000 7f2ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.536+0000 7f2ccaffd640 10 monclient(hunting): my global_id is 4241 2024-03-25T19:41:35.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.536+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb8004eb0 con 0x7f2cd41430c0 2024-03-25T19:41:35.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.536+0000 7f2ccaffd640 10 client.?.objecter ms_handle_connect 0x7f2cd40a98f0 2024-03-25T19:41:35.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.536+0000 7f2ccaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:35.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680561644 0 0) 0x7f2cd41d7380 con 0x7f2cd40a98f0 2024-03-25T19:41:35.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:41:35.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd41d8200 con 0x7f2cd40a98f0 2024-03-25T19:41:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599687950 0 0) 0x7f2cd4139840 con 0x7f2cd413f710 2024-03-25T19:41:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 10 monclient(hunting): my global_id is 4237 2024-03-25T19:41:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.548+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd41d7380 con 0x7f2cd413f710 2024-03-25T19:41:35.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.554+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3008127040 0 0) 0x7f2cb8004eb0 con 0x7f2cd41430c0 2024-03-25T19:41:35.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.555+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cd4139840 con 0x7f2cd41430c0 2024-03-25T19:41:35.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.555+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2ccc001c30 con 0x7f2cd41430c0 2024-03-25T19:41:35.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.555+0000 7f2ccaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.555+0000 7f2ccaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:35.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.555+0000 7f2ccaffd640 10 monclient(hunting): dump: 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:35.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.555+0000 7f2ccaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:35.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.577+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2602566009 0 0) 0x7f2cd41d8200 con 0x7f2cd40a98f0 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.577+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb8004eb0 con 0x7f2cd40a98f0 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.577+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2cd0001b40 con 0x7f2cd40a98f0 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.577+0000 7f2ccaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.577+0000 7f2ccaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.577+0000 7f2ccaffd640 10 monclient(hunting): dump: 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:35.577 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:35.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2231783487 0 0) 0x7f2cd4139840 con 0x7f2cd41430c0 2024-03-25T19:41:35.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 >> v1:172.21.15.53:6789/0 conn(0x7f2cd413f710 legacy=0x7f2cd41da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 >> v1:172.21.15.18:6789/0 conn(0x7f2cd40a98f0 legacy=0x7f2cd41d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 1 monclient: found mon.c 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd41d75d0 con 0x7f2cd41430c0 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2ccaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 5 monclient: authenticate success, global_id 4241 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 10 monclient: _renew_subs 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd41db6a0 con 0x7f2cd41430c0 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 10 monclient: _renew_subs 2024-03-25T19:41:35.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:35.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd4137910 con 0x7f2cd41430c0 2024-03-25T19:41:35.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.582+0000 7f2cdc4d3640 1 librados: init done 2024-03-25T19:41:35.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.583+0000 7f2cdc4d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:35.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.584+0000 7f2cdc4d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:35.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.584+0000 7f2cdc4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:35.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.584+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd40627b0 con 0x7f2cd41430c0 2024-03-25T19:41:35.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.906+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ccc003990 con 0x7f2cd41430c0 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.906+0000 7f2ccaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.906+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2ccc003d20 con 0x7f2cd41430c0 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.906+0000 7f2ccaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.906+0000 7f2ccaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.906+0000 7f2ccaffd640 10 monclient: dump: 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:35.906 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:35.907 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:35.907 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:35.907 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:35.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.906+0000 7f2cd8a6c640 4 set_mon_vals no callback set 2024-03-25T19:41:35.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.907+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f2ccc003f40 con 0x7f2cd41430c0 2024-03-25T19:41:35.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.907+0000 7f2cda26f640 10 monclient: get_auth_request con 0x7f2cb806df70 auth_method 0 2024-03-25T19:41:35.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.908+0000 7f2ccaffd640 10 client.4241.objecter ms_handle_connect 0x7f2cb806df70 2024-03-25T19:41:35.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.913+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f2ccc054a40 con 0x7f2cd41430c0 2024-03-25T19:41:35.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.913+0000 7f2ccaffd640 10 client.4241.objecter ms_dispatch 0x7f2cd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:35.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.913+0000 7f2ccaffd640 3 client.4241.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:35.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.913+0000 7f2ccaffd640 3 client.4241.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:35.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.913+0000 7f2ccaffd640 20 client.4241.objecter dump_active .. 0 homeless 2024-03-25T19:41:35.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.913+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ccc050550 con 0x7f2cd41430c0 2024-03-25T19:41:35.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.913+0000 7f2ccaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:35.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:35.913+0000 7f2ccaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:36.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.137+0000 7f2cdc4d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.137+0000 7f2cdc4d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.137+0000 7f2cdc4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:36.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.137+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cd4069f40 con 0x7f2cd41430c0 2024-03-25T19:41:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.208+0000 7f2ccaffd640 1 -- 172.21.15.18:0/2815830899 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f2ccc052480 con 0x7f2cd41430c0 2024-03-25T19:41:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.208+0000 7f2ccaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.208+0000 7f2ccaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:36.208 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:41:36.208 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:36.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.210+0000 7f2cdc4d3640 10 librados: watch_flush enter 2024-03-25T19:41:36.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.210+0000 7f2cdc4d3640 10 librados: watch_flush exit 2024-03-25T19:41:36.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.210+0000 7f2cdc4d3640 20 client.4241.objecter shutdown clearing up homeless session... 2024-03-25T19:41:36.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.210+0000 7f2cdc4d3640 10 client.4241.objecter successfully canceled tick 2024-03-25T19:41:36.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.210+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 >> v1:172.21.15.18:6800/1520000573 conn(0x7f2cb806df70 legacy=0x7f2cb8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.210+0000 7f2cdc4d3640 10 monclient: shutdown 2024-03-25T19:41:36.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.210+0000 7f2cdc4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:36.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.210+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 >> v1:172.21.15.18:6790/0 conn(0x7f2cd41430c0 legacy=0x7f2cd41daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.211+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 shutdown_connections 2024-03-25T19:41:36.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.211+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 >> 172.21.15.18:0/2815830899 conn(0x7f2cd40b7ea0 msgr2=0x7f2cd4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:36.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.211+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 shutdown_connections 2024-03-25T19:41:36.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.211+0000 7f2cdc4d3640 1 -- 172.21.15.18:0/2815830899 wait complete. 2024-03-25T19:41:36.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:36.211+0000 7f2cdc4d3640 1 librados: shutdown 2024-03-25T19:41:36.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:43.030 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.145+0000 7f4b43515640 10 monclient: get_monmap_and_config 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.145+0000 7f4b43515640 10 monclient: build_initial_monmap 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.145+0000 7f4b43515640 10 monclient: monmap: 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:41:43.146339+0000 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:41:43.146339+0000 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:41:43.146 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.146+0000 7f4b43515640 1 Processor -- start 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.146+0000 7f4b43515640 1 -- start start 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.146+0000 7f4b43515640 10 monclient: init 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 10 monclient(hunting): picked mon.noname-b con 0x7f4b3c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 10 monclient(hunting): picked mon.noname-c con 0x7f4b3c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 10 monclient(hunting): picked mon.noname-a con 0x7f4b3c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c142520 con 0x7f4b3c1430c0 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0b28a0 con 0x7f4b3c13f710 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0b2a30 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.147+0000 7f4b43515640 10 monclient(hunting): authenticate will time out at 4930.273997s 2024-03-25T19:41:43.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.157+0000 7f4b41ab2640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f4b3c1430c0 0x7f4b3c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:36714/0 (socket says 172.21.15.18:36714) 2024-03-25T19:41:43.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.157+0000 7f4b41ab2640 1 -- 172.21.15.18:0/756360702 learned_addr learned my addr 172.21.15.18:0/756360702 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:43.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.178+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352361172 0 0) 0x7f4b3c0b2a30 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.178+0000 7f4b33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.178+0000 7f4b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.178+0000 7f4b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.178+0000 7f4b33fff640 10 monclient(hunting): my global_id is 4244 2024-03-25T19:41:43.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.178+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b28003520 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589116714 0 0) 0x7f4b3c0b28a0 con 0x7f4b3c13f710 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 10 monclient(hunting): my global_id is 4240 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c0b2a30 con 0x7f4b3c13f710 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799595866 0 0) 0x7f4b3c142520 con 0x7f4b3c1430c0 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:41:43.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.181+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c0b28a0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 986997050 0 0) 0x7f4b3c0b28a0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 >> v1:172.21.15.18:6790/0 conn(0x7f4b3c0a98f0 legacy=0x7f4b3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 >> v1:172.21.15.53:6789/0 conn(0x7f4b3c13f710 legacy=0x7f4b3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 1 monclient: found mon.noname-a 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0b44c0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4b34001930 con 0x7f4b3c1430c0 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b33fff640 10 monclient: dump: 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b43515640 5 monclient: authenticate success, global_id 4260 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.188+0000 7f4b43515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:43.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.213+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b34001a90 con 0x7f4b3c1430c0 2024-03-25T19:41:43.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.213+0000 7f4b33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:43.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.213+0000 7f4b33fff640 1 -- 172.21.15.18:0/756360702 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4b34001ea0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.213+0000 7f4b33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.213+0000 7f4b33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.213+0000 7f4b33fff640 10 monclient: dump: 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:43.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.213+0000 7f4b43515640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:43.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.213+0000 7f4b43515640 4 set_mon_vals no callback set 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.214+0000 7f4b43515640 10 monclient: shutdown 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.214+0000 7f4b43515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.214+0000 7f4b43515640 1 -- 172.21.15.18:0/756360702 >> v1:172.21.15.18:6789/0 conn(0x7f4b3c1430c0 legacy=0x7f4b3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.214+0000 7f4b43515640 1 -- 172.21.15.18:0/756360702 shutdown_connections 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.214+0000 7f4b43515640 1 -- 172.21.15.18:0/756360702 >> 172.21.15.18:0/756360702 conn(0x7f4b3c0b7ea0 msgr2=0x7f4b3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.214+0000 7f4b43515640 1 -- 172.21.15.18:0/756360702 shutdown_connections 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.214+0000 7f4b43515640 1 -- 172.21.15.18:0/756360702 wait complete. 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 10 monclient: build_initial_monmap 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 10 monclient: monmap: 2024-03-25T19:41:43.215 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 1 librados: starting msgr at 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 1 librados: starting objecter 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 1 Processor -- start 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 1 -- start start 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 1 librados: setting wanted keys 2024-03-25T19:41:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 1 librados: calling monclient init 2024-03-25T19:41:43.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.215+0000 7f4b43515640 10 monclient: init 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.216+0000 7f4b43515640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.216+0000 7f4b43515640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.216+0000 7f4b43515640 10 monclient(hunting): picked mon.noname-a con 0x7f4b3c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.216+0000 7f4b43515640 10 monclient(hunting): picked mon.noname-b con 0x7f4b3c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.217+0000 7f4b43515640 10 monclient(hunting): picked mon.noname-c con 0x7f4b3c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.217+0000 7f4b43515640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0abcf0 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.217+0000 7f4b43515640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c1d51f0 con 0x7f4b3c13f710 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.217+0000 7f4b43515640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c1d83d0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.217+0000 7f4b43515640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.217+0000 7f4b43515640 10 monclient(hunting): authenticate will time out at 4930.343893s 2024-03-25T19:41:43.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.224+0000 7f4b41ab2640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f4b3c1430c0 0x7f4b3c1e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46380/0 (socket says 172.21.15.18:46380) 2024-03-25T19:41:43.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.224+0000 7f4b41ab2640 1 -- 172.21.15.18:0/2474641847 learned_addr learned my addr 172.21.15.18:0/2474641847 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:43.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.225+0000 7f4b31ffb640 10 client.?.objecter ms_handle_connect 0x7f4b3c1430c0 2024-03-25T19:41:43.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.225+0000 7f4b31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.228+0000 7f4b31ffb640 10 client.?.objecter ms_handle_connect 0x7f4b3c0a98f0 2024-03-25T19:41:43.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.228+0000 7f4b31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.233+0000 7f4b31ffb640 10 client.?.objecter ms_handle_connect 0x7f4b3c13f710 2024-03-25T19:41:43.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.233+0000 7f4b31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.242+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315582165 0 0) 0x7f4b3c1d83d0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.242+0000 7f4b31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.242+0000 7f4b31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.242+0000 7f4b31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.242+0000 7f4b31ffb640 10 monclient(hunting): my global_id is 4247 2024-03-25T19:41:43.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.242+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b20004eb0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.246+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323150030 0 0) 0x7f4b3c0abcf0 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.246+0000 7f4b31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.246+0000 7f4b31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.246+0000 7f4b31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.246+0000 7f4b31ffb640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:41:43.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.246+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c1d83d0 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.254+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753059011 0 0) 0x7f4b3c1d51f0 con 0x7f4b3c13f710 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.254+0000 7f4b31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.254+0000 7f4b31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.254+0000 7f4b31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.254+0000 7f4b31ffb640 10 monclient(hunting): my global_id is 4243 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.254+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c0abcf0 con 0x7f4b3c13f710 2024-03-25T19:41:43.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.265+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3639865245 0 0) 0x7f4b3c1d83d0 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.265+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b3c1d51f0 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.265+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4b38001950 con 0x7f4b3c0a98f0 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.265+0000 7f4b31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.265+0000 7f4b31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.265+0000 7f4b31ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:43.265 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:43.266 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:43.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.265+0000 7f4b31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:43.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.270+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3696710860 0 0) 0x7f4b20004eb0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.270+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b3c1d83d0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.270+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4b34001ea0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.270+0000 7f4b31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.270+0000 7f4b31ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.270+0000 7f4b31ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:43.270 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:43.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.278+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3735880878 0 0) 0x7f4b3c0abcf0 con 0x7f4b3c13f710 2024-03-25T19:41:43.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.278+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b20004eb0 con 0x7f4b3c13f710 2024-03-25T19:41:43.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.278+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4b2c001930 con 0x7f4b3c13f710 2024-03-25T19:41:43.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.278+0000 7f4b31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.278+0000 7f4b31ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.278+0000 7f4b31ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:43.278 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1314596100 0 0) 0x7f4b3c1d83d0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 >> v1:172.21.15.53:6789/0 conn(0x7f4b3c13f710 legacy=0x7f4b3c138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 >> v1:172.21.15.18:6789/0 conn(0x7f4b3c0a98f0 legacy=0x7f4b3c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 1 monclient: found mon.c 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c138e40 con 0x7f4b3c1430c0 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b43515640 5 monclient: authenticate success, global_id 4247 2024-03-25T19:41:43.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b43515640 10 monclient: _renew_subs 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b43515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.279+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c137f70 con 0x7f4b3c1430c0 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.280+0000 7f4b43515640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.280+0000 7f4b43515640 10 monclient: _renew_subs 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.280+0000 7f4b43515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.280+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c1e3550 con 0x7f4b3c1430c0 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.280+0000 7f4b43515640 1 librados: init done 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.281+0000 7f4b43515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.281+0000 7f4b43515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.281+0000 7f4b43515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:43.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.281+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c1d7600 con 0x7f4b3c1430c0 2024-03-25T19:41:43.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.495+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b34004e10 con 0x7f4b3c1430c0 2024-03-25T19:41:43.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.495+0000 7f4b31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:43.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.495+0000 7f4b337fe640 4 set_mon_vals no callback set 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.602+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4b34002100 con 0x7f4b3c1430c0 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.602+0000 7f4b31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.602+0000 7f4b31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.602+0000 7f4b31ffb640 10 monclient: dump: 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.602 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:43.603 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:43.603 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:43.603 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:43.603 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:43.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.603+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f4b3401adf0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.604+0000 7f4b412b1640 10 monclient: get_auth_request con 0x7f4b2006e160 auth_method 0 2024-03-25T19:41:43.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.604+0000 7f4b31ffb640 10 client.4247.objecter ms_handle_connect 0x7f4b2006e160 2024-03-25T19:41:43.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.609+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f4b340549f0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.609+0000 7f4b31ffb640 10 client.4247.objecter ms_dispatch 0x7f4b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:43.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.609+0000 7f4b31ffb640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:43.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.609+0000 7f4b31ffb640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:43.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.609+0000 7f4b31ffb640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-25T19:41:43.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.609+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b34050330 con 0x7f4b3c1430c0 2024-03-25T19:41:43.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.610+0000 7f4b31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.610+0000 7f4b31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:43.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.855+0000 7f4b43515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:43.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.855+0000 7f4b43515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:43.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.855+0000 7f4b43515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:41:43.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.855+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b3c0627b0 con 0x7f4b3c1430c0 2024-03-25T19:41:43.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.880+0000 7f4b31ffb640 1 -- 172.21.15.18:0/2474641847 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f4b34052430 con 0x7f4b3c1430c0 2024-03-25T19:41:43.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.880+0000 7f4b31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:43.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.880+0000 7f4b31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:43.879 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:41:43.880 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:43.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.881+0000 7f4b43515640 10 librados: watch_flush enter 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.881+0000 7f4b43515640 10 librados: watch_flush exit 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.881+0000 7f4b43515640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.881+0000 7f4b43515640 10 client.4247.objecter successfully canceled tick 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 >> v1:172.21.15.18:6800/1520000573 conn(0x7f4b2006e160 legacy=0x7f4b20070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 10 monclient: shutdown 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 >> v1:172.21.15.18:6790/0 conn(0x7f4b3c1430c0 legacy=0x7f4b3c1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 shutdown_connections 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 >> 172.21.15.18:0/2474641847 conn(0x7f4b3c0b7ea0 msgr2=0x7f4b3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 shutdown_connections 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 1 -- 172.21.15.18:0/2474641847 wait complete. 2024-03-25T19:41:43.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:43.882+0000 7f4b43515640 1 librados: shutdown 2024-03-25T19:41:43.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:46.843 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:14.374 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:46.921 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:14.452 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:46.987 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:14.518 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:47.054 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:14.585 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:47.135 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:14.632 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:47.196 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:14.727 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:47.202 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:14.698 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:47.289 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:14.820 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:47.439 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:14.936 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:50.700 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:50.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.813+0000 7f1155a4d640 10 monclient: get_monmap_and_config 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.813+0000 7f1155a4d640 10 monclient: build_initial_monmap 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.813+0000 7f1155a4d640 10 monclient: monmap: 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:41:50.813917+0000 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:41:50.813917+0000 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:41:50.813 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:50.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.813+0000 7f1155a4d640 1 Processor -- start 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.813+0000 7f1155a4d640 1 -- start start 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.813+0000 7f1155a4d640 10 monclient: init 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 10 monclient(hunting): picked mon.noname-a con 0x7f11500a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 10 monclient(hunting): picked mon.noname-c con 0x7f115013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 10 monclient(hunting): picked mon.noname-b con 0x7f11501430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1150142520 con 0x7f11500a98f0 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500b28a0 con 0x7f115013f710 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11500b2a30 con 0x7f11501430c0 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.814+0000 7f1155a4d640 10 monclient(hunting): authenticate will time out at 4937.941460s 2024-03-25T19:41:50.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.816+0000 7f114e7fc640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f115013f710 0x7f11501429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:48718/0 (socket says 172.21.15.18:48718) 2024-03-25T19:41:50.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.816+0000 7f114e7fc640 1 -- 172.21.15.18:0/27330849 learned_addr learned my addr 172.21.15.18:0/27330849 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:50.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.827+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555128722 0 0) 0x7f11500b28a0 con 0x7f115013f710 2024-03-25T19:41:50.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.827+0000 7f114dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.827+0000 7f114dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.827+0000 7f114dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.827+0000 7f114dffb640 10 monclient(hunting): my global_id is 4246 2024-03-25T19:41:50.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.827+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1138003520 con 0x7f115013f710 2024-03-25T19:41:50.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.833+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598127930 0 0) 0x7f1150142520 con 0x7f11500a98f0 2024-03-25T19:41:50.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.833+0000 7f114dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.833+0000 7f114dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.833+0000 7f114dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.833+0000 7f114dffb640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:41:50.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.833+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11500b28a0 con 0x7f11500a98f0 2024-03-25T19:41:50.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.838+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463256763 0 0) 0x7f11500b2a30 con 0x7f11501430c0 2024-03-25T19:41:50.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.838+0000 7f114dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.838+0000 7f114dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.838+0000 7f114dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.838+0000 7f114dffb640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:41:50.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.838+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1150142520 con 0x7f11501430c0 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902788331 0 0) 0x7f1150142520 con 0x7f11501430c0 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 >> v1:172.21.15.53:6789/0 conn(0x7f115013f710 legacy=0x7f11501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 >> v1:172.21.15.18:6789/0 conn(0x7f11500a98f0 legacy=0x7f11500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 1 monclient: found mon.noname-b 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11500b44c0 con 0x7f11501430c0 2024-03-25T19:41:50.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1148001930 con 0x7f11501430c0 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f114dffb640 10 monclient: dump: 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:50.848 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:50.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f1155a4d640 5 monclient: authenticate success, global_id 4250 2024-03-25T19:41:50.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.847+0000 7f1155a4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:50.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.865+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1148001a90 con 0x7f11501430c0 2024-03-25T19:41:50.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.865+0000 7f114dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:50.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.865+0000 7f114dffb640 1 -- 172.21.15.18:0/27330849 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1148001ee0 con 0x7f11501430c0 2024-03-25T19:41:50.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.865+0000 7f1155a4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:50.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.865+0000 7f1155a4d640 4 set_mon_vals no callback set 2024-03-25T19:41:50.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f114dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f114dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f114dffb640 10 monclient: dump: 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:50.866 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f1155a4d640 10 monclient: shutdown 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f1155a4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f1155a4d640 1 -- 172.21.15.18:0/27330849 >> v1:172.21.15.18:6790/0 conn(0x7f11501430c0 legacy=0x7f115014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f1155a4d640 1 -- 172.21.15.18:0/27330849 shutdown_connections 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f1155a4d640 1 -- 172.21.15.18:0/27330849 >> 172.21.15.18:0/27330849 conn(0x7f11500b7ea0 msgr2=0x7f11500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f1155a4d640 1 -- 172.21.15.18:0/27330849 shutdown_connections 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.866+0000 7f1155a4d640 1 -- 172.21.15.18:0/27330849 wait complete. 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.867+0000 7f1155a4d640 10 monclient: build_initial_monmap 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.867+0000 7f1155a4d640 10 monclient: monmap: 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:41:50.869 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.867+0000 7f1155a4d640 1 librados: starting msgr at 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.867+0000 7f1155a4d640 1 librados: starting objecter 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.867+0000 7f1155a4d640 1 Processor -- start 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 1 -- start start 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 1 librados: setting wanted keys 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 1 librados: calling monclient init 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 10 monclient: init 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:50.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:50.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 10 monclient(hunting): picked mon.noname-c con 0x7f11500a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:50.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 10 monclient(hunting): picked mon.noname-a con 0x7f115013f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:50.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 10 monclient(hunting): picked mon.noname-b con 0x7f11501430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:50.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115014e4e0 con 0x7f115013f710 2024-03-25T19:41:50.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115006fd60 con 0x7f11501430c0 2024-03-25T19:41:50.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1150070040 con 0x7f11500a98f0 2024-03-25T19:41:50.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:50.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.868+0000 7f1155a4d640 10 monclient(hunting): authenticate will time out at 4937.995674s 2024-03-25T19:41:50.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.881+0000 7f114f7fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f11501430c0 0x7f11501e64e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:48776/0 (socket says 172.21.15.18:48776) 2024-03-25T19:41:50.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.881+0000 7f114f7fe640 1 -- 172.21.15.18:0/352318106 learned_addr learned my addr 172.21.15.18:0/352318106 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:50.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.885+0000 7f1137fff640 10 client.?.objecter ms_handle_connect 0x7f11501430c0 2024-03-25T19:41:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.885+0000 7f1137fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.895+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021406270 0 0) 0x7f115006fd60 con 0x7f11501430c0 2024-03-25T19:41:50.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.895+0000 7f1137fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.895+0000 7f1137fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.895+0000 7f1137fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.895+0000 7f1137fff640 10 monclient(hunting): my global_id is 4249 2024-03-25T19:41:50.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.895+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c004eb0 con 0x7f11501430c0 2024-03-25T19:41:50.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.900+0000 7f1137fff640 10 client.?.objecter ms_handle_connect 0x7f11500a98f0 2024-03-25T19:41:50.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.900+0000 7f1137fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.905+0000 7f1137fff640 10 client.?.objecter ms_handle_connect 0x7f115013f710 2024-03-25T19:41:50.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.905+0000 7f1137fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.908+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3088848888 0 0) 0x7f113c004eb0 con 0x7f11501430c0 2024-03-25T19:41:50.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.908+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f115006fd60 con 0x7f11501430c0 2024-03-25T19:41:50.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.908+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1148001ea0 con 0x7f11501430c0 2024-03-25T19:41:50.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.908+0000 7f1137fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.908+0000 7f1137fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:50.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.908+0000 7f1137fff640 10 monclient(hunting): dump: 2024-03-25T19:41:50.908 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:50.908 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:50.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.908+0000 7f1137fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:50.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.914+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421849608 0 0) 0x7f115014e4e0 con 0x7f115013f710 2024-03-25T19:41:50.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.914+0000 7f1137fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.914+0000 7f1137fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.914+0000 7f1137fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.914+0000 7f1137fff640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:41:50.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.914+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c004eb0 con 0x7f115013f710 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2697211950 0 0) 0x7f115006fd60 con 0x7f11501430c0 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 >> v1:172.21.15.18:6790/0 conn(0x7f11500a98f0 legacy=0x7f115014bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 >> v1:172.21.15.18:6789/0 conn(0x7f115013f710 legacy=0x7f115014c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 1 monclient: found mon.b 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115006fa80 con 0x7f11501430c0 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:50.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1137fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 5 monclient: authenticate success, global_id 4249 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 10 monclient: _renew_subs 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115006ec70 con 0x7f11501430c0 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 10 monclient: _renew_subs 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f115006e2f0 con 0x7f11501430c0 2024-03-25T19:41:50.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.924+0000 7f1155a4d640 1 librados: init done 2024-03-25T19:41:50.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.925+0000 7f1155a4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:50.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.925+0000 7f1155a4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:50.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.925+0000 7f1155a4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:50.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:50.925+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11500aa500 con 0x7f11501430c0 2024-03-25T19:41:51.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.229+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1148003c20 con 0x7f11501430c0 2024-03-25T19:41:51.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.229+0000 7f1137fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:51.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.229+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1148001590 con 0x7f11501430c0 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.229+0000 7f1137fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.229+0000 7f1137fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.229+0000 7f1137fff640 10 monclient: dump: 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:51.229 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:51.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.229+0000 7f114cff9640 4 set_mon_vals no callback set 2024-03-25T19:41:51.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.229+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f11480040b0 con 0x7f11501430c0 2024-03-25T19:41:51.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.230+0000 7f114effd640 10 monclient: get_auth_request con 0x7f113c06dd90 auth_method 0 2024-03-25T19:41:51.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.230+0000 7f1137fff640 10 client.4249.objecter ms_handle_connect 0x7f113c06dd90 2024-03-25T19:41:51.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.235+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f1148054ad0 con 0x7f11501430c0 2024-03-25T19:41:51.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.235+0000 7f1137fff640 10 client.4249.objecter ms_dispatch 0x7f1150000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:51.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.235+0000 7f1137fff640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:51.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.235+0000 7f1137fff640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:51.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.235+0000 7f1137fff640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-25T19:41:51.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.238+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11480505e0 con 0x7f11501430c0 2024-03-25T19:41:51.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.238+0000 7f1137fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:51.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.238+0000 7f1137fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:51.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.496+0000 7f1155a4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:51.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.496+0000 7f1155a4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:51.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.496+0000 7f1155a4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:51.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.496+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f115014f2d0 con 0x7f11501430c0 2024-03-25T19:41:51.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.527+0000 7f1137fff640 1 -- 172.21.15.18:0/352318106 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f1148052510 con 0x7f11501430c0 2024-03-25T19:41:51.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.527+0000 7f1137fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:51.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.527+0000 7f1137fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:51.527 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:41:51.527 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 10 librados: watch_flush enter 2024-03-25T19:41:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 10 librados: watch_flush exit 2024-03-25T19:41:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-25T19:41:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 10 client.4249.objecter successfully canceled tick 2024-03-25T19:41:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 >> v1:172.21.15.18:6800/1520000573 conn(0x7f113c06dd90 legacy=0x7f113c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 10 monclient: shutdown 2024-03-25T19:41:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 >> v1:172.21.15.53:6789/0 conn(0x7f11501430c0 legacy=0x7f11501e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 shutdown_connections 2024-03-25T19:41:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 >> 172.21.15.18:0/352318106 conn(0x7f11500b7ea0 msgr2=0x7f1150074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 shutdown_connections 2024-03-25T19:41:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 1 -- 172.21.15.18:0/352318106 wait complete. 2024-03-25T19:41:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:51.530+0000 7f1155a4d640 1 librados: shutdown 2024-03-25T19:41:51.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:58.348 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.463+0000 7f782ba06640 10 monclient: get_monmap_and_config 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.463+0000 7f782ba06640 10 monclient: build_initial_monmap 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.463+0000 7f782ba06640 10 monclient: monmap: 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:41:58.464280+0000 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:41:58.464280+0000 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.464+0000 7f782ba06640 1 Processor -- start 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.464+0000 7f782ba06640 1 -- start start 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.464+0000 7f782ba06640 10 monclient: init 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 10 monclient(hunting): picked mon.noname-a con 0x7f78240a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 10 monclient(hunting): picked mon.noname-c con 0x7f782413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 10 monclient(hunting): picked mon.noname-b con 0x7f78241430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7824142520 con 0x7f78240a98f0 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78240b28a0 con 0x7f782413f710 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78240b2a30 con 0x7f78241430c0 2024-03-25T19:41:58.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:58.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.465+0000 7f782ba06640 10 monclient(hunting): authenticate will time out at 4945.592106s 2024-03-25T19:41:58.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.477+0000 7f7828fa1640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f782413f710 0x7f78241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:36258/0 (socket says 172.21.15.18:36258) 2024-03-25T19:41:58.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.477+0000 7f7828fa1640 1 -- 172.21.15.18:0/1274986098 learned_addr learned my addr 172.21.15.18:0/1274986098 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:58.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.483+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387132087 0 0) 0x7f78240b28a0 con 0x7f782413f710 2024-03-25T19:41:58.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.483+0000 7f781bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.483+0000 7f781bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.483+0000 7f781bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.483+0000 7f781bfff640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:41:58.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.483+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7810003520 con 0x7f782413f710 2024-03-25T19:41:58.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.489+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411627255 0 0) 0x7f78240b2a30 con 0x7f78241430c0 2024-03-25T19:41:58.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.489+0000 7f781bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.489+0000 7f781bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.489+0000 7f781bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.489+0000 7f781bfff640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:41:58.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.489+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78240b28a0 con 0x7f78241430c0 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.493+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090856459 0 0) 0x7f7824142520 con 0x7f78240a98f0 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.493+0000 7f781bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.493+0000 7f781bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.493+0000 7f781bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.493+0000 7f781bfff640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:41:58.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.493+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78240b2a30 con 0x7f78240a98f0 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159084514 0 0) 0x7f78240b28a0 con 0x7f78241430c0 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 >> v1:172.21.15.53:6789/0 conn(0x7f782413f710 legacy=0x7f78241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 >> v1:172.21.15.18:6789/0 conn(0x7f78240a98f0 legacy=0x7f78240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 1 monclient: found mon.noname-b 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78240b44c0 con 0x7f78241430c0 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:58.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f781c0019f0 con 0x7f78241430c0 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f781bfff640 10 monclient: dump: 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f782ba06640 5 monclient: authenticate success, global_id 4256 2024-03-25T19:41:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.499+0000 7f782ba06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:58.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.510+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f781c001b50 con 0x7f78241430c0 2024-03-25T19:41:58.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.510+0000 7f781bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:58.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.510+0000 7f781bfff640 1 -- 172.21.15.18:0/1274986098 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f781c001f60 con 0x7f78241430c0 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.510+0000 7f781bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.510+0000 7f781bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.510+0000 7f781bfff640 10 monclient: dump: 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:58.510 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:58.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.510+0000 7f782ba06640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:58.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.510+0000 7f782ba06640 4 set_mon_vals no callback set 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.511+0000 7f782ba06640 10 monclient: shutdown 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.511+0000 7f782ba06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.511+0000 7f782ba06640 1 -- 172.21.15.18:0/1274986098 >> v1:172.21.15.18:6790/0 conn(0x7f78241430c0 legacy=0x7f782414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.511+0000 7f782ba06640 1 -- 172.21.15.18:0/1274986098 shutdown_connections 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.511+0000 7f782ba06640 1 -- 172.21.15.18:0/1274986098 >> 172.21.15.18:0/1274986098 conn(0x7f78240b7ea0 msgr2=0x7f78240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.511+0000 7f782ba06640 1 -- 172.21.15.18:0/1274986098 shutdown_connections 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.511+0000 7f782ba06640 1 -- 172.21.15.18:0/1274986098 wait complete. 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 10 monclient: build_initial_monmap 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 10 monclient: monmap: 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:41:58.513 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 1 librados: starting msgr at 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 1 librados: starting objecter 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 1 Processor -- start 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 1 -- start start 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 1 librados: setting wanted keys 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 1 librados: calling monclient init 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.512+0000 7f782ba06640 10 monclient: init 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 10 monclient(hunting): picked mon.noname-b con 0x7f78240a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 10 monclient(hunting): picked mon.noname-a con 0x7f782413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 10 monclient(hunting): picked mon.noname-c con 0x7f78241430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78241397d0 con 0x7f782413f710 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782413af80 con 0x7f78240a98f0 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78241dba00 con 0x7f78241430c0 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:58.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.513+0000 7f782ba06640 10 monclient(hunting): authenticate will time out at 4945.640019s 2024-03-25T19:41:58.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.521+0000 7f78297a2640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f78240a98f0 0x7f78240b6de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:36302/0 (socket says 172.21.15.18:36302) 2024-03-25T19:41:58.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.521+0000 7f78297a2640 1 -- 172.21.15.18:0/1278722576 learned_addr learned my addr 172.21.15.18:0/1278722576 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:41:58.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.523+0000 7f7819ffb640 10 client.?.objecter ms_handle_connect 0x7f78240a98f0 2024-03-25T19:41:58.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.523+0000 7f7819ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:58.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.532+0000 7f7819ffb640 10 client.?.objecter ms_handle_connect 0x7f78241430c0 2024-03-25T19:41:58.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.532+0000 7f7819ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:58.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.535+0000 7f7819ffb640 10 client.?.objecter ms_handle_connect 0x7f782413f710 2024-03-25T19:41:58.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.535+0000 7f7819ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:58.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.537+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373087176 0 0) 0x7f782413af80 con 0x7f78240a98f0 2024-03-25T19:41:58.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.537+0000 7f7819ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.537+0000 7f7819ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.537+0000 7f7819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.537+0000 7f7819ffb640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:41:58.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.537+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7808004eb0 con 0x7f78240a98f0 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.541+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110213213 0 0) 0x7f78241397d0 con 0x7f782413f710 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.541+0000 7f7819ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.541+0000 7f7819ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.541+0000 7f7819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.541+0000 7f7819ffb640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.541+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f782413af80 con 0x7f782413f710 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.542+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732011171 0 0) 0x7f78241dba00 con 0x7f78241430c0 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.542+0000 7f7819ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.542+0000 7f7819ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.542+0000 7f7819ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.542+0000 7f7819ffb640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.542+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78241397d0 con 0x7f78241430c0 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.555+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3738300589 0 0) 0x7f7808004eb0 con 0x7f78240a98f0 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.555+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78241dba00 con 0x7f78240a98f0 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.555+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7820001a10 con 0x7f78240a98f0 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.555+0000 7f7819ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.555+0000 7f7819ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.555+0000 7f7819ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.555 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.556 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.556 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:58.556 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:58.556 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:58.556 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:58.556 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:58.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.555+0000 7f7819ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.559+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2283137796 0 0) 0x7f782413af80 con 0x7f782413f710 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.559+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7808004eb0 con 0x7f782413f710 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.559+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7814001950 con 0x7f782413f710 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.559+0000 7f7819ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.559+0000 7f7819ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.559+0000 7f7819ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.559 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.560 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:58.560 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:58.560 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:58.560 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:58.560 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:58.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.560+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3945879154 0 0) 0x7f78241397d0 con 0x7f78241430c0 2024-03-25T19:41:58.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.560+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f782413af80 con 0x7f78241430c0 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.560+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f781c001e60 con 0x7f78241430c0 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.560+0000 7f7819ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.560+0000 7f7819ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.560+0000 7f7819ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:58.561 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:58.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 401126048 0 0) 0x7f78241dba00 con 0x7f78240a98f0 2024-03-25T19:41:58.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 >> v1:172.21.15.18:6790/0 conn(0x7f78241430c0 legacy=0x7f78241db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 >> v1:172.21.15.18:6789/0 conn(0x7f782413f710 legacy=0x7f78241dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 1 monclient: found mon.b 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782413b260 con 0x7f78240a98f0 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f7819ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 5 monclient: authenticate success, global_id 4255 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 10 monclient: _renew_subs 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78240b74f0 con 0x7f78240a98f0 2024-03-25T19:41:58.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 10 monclient: _renew_subs 2024-03-25T19:41:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f782412f830 con 0x7f78240a98f0 2024-03-25T19:41:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.574+0000 7f782ba06640 1 librados: init done 2024-03-25T19:41:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.575+0000 7f782ba06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.575+0000 7f782ba06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.575+0000 7f782ba06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.575+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78240627b0 con 0x7f78240a98f0 2024-03-25T19:41:58.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.869+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7820001c70 con 0x7f78240a98f0 2024-03-25T19:41:58.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.869+0000 7f7819ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:58.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.869+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7820003a60 con 0x7f78240a98f0 2024-03-25T19:41:58.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.869+0000 7f7819ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.869+0000 7f7819ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:58.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.869+0000 7f7819ffb640 10 monclient: dump: 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:41:58.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.869+0000 7f781b7fe640 4 set_mon_vals no callback set 2024-03-25T19:41:58.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.870+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f7820003c80 con 0x7f78240a98f0 2024-03-25T19:41:58.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.871+0000 7f7828fa1640 10 monclient: get_auth_request con 0x7f780806e0f0 auth_method 0 2024-03-25T19:41:58.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.871+0000 7f7819ffb640 10 client.4255.objecter ms_handle_connect 0x7f780806e0f0 2024-03-25T19:41:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.875+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f7820054860 con 0x7f78240a98f0 2024-03-25T19:41:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.875+0000 7f7819ffb640 10 client.4255.objecter ms_dispatch 0x7f7824000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.875+0000 7f7819ffb640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.875+0000 7f7819ffb640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.875+0000 7f7819ffb640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-25T19:41:58.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.877+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7820050370 con 0x7f78240a98f0 2024-03-25T19:41:58.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.877+0000 7f7819ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:58.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:58.877+0000 7f7819ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:59.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.120+0000 7f782ba06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.120+0000 7f782ba06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.120+0000 7f782ba06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:41:59.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.120+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f782414e8f0 con 0x7f78240a98f0 2024-03-25T19:41:59.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.165+0000 7f7819ffb640 1 -- 172.21.15.18:0/1278722576 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f78200522a0 con 0x7f78240a98f0 2024-03-25T19:41:59.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.165+0000 7f7819ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.165+0000 7f7819ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:59.165 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:41:59.165 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:59.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.167+0000 7f782ba06640 10 librados: watch_flush enter 2024-03-25T19:41:59.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.167+0000 7f782ba06640 10 librados: watch_flush exit 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.167+0000 7f782ba06640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.167+0000 7f782ba06640 10 client.4255.objecter successfully canceled tick 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.167+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 >> v1:172.21.15.18:6800/1520000573 conn(0x7f780806e0f0 legacy=0x7f78080705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.167+0000 7f782ba06640 10 monclient: shutdown 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.167+0000 7f782ba06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.167+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 >> v1:172.21.15.53:6789/0 conn(0x7f78240a98f0 legacy=0x7f78240b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.168+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 shutdown_connections 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.168+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 >> 172.21.15.18:0/1278722576 conn(0x7f78240b7ea0 msgr2=0x7f7824074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.168+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 shutdown_connections 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.168+0000 7f782ba06640 1 -- 172.21.15.18:0/1278722576 wait complete. 2024-03-25T19:41:59.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:41:59.168+0000 7f782ba06640 1 librados: shutdown 2024-03-25T19:41:59.180 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:05.986 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:06.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.094+0000 7fef5c08d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:06.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.094+0000 7fef5c08d640 10 monclient: build_initial_monmap 2024-03-25T19:42:06.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.094+0000 7fef5c08d640 10 monclient: monmap: 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:42:06.095718+0000 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:42:06.095718+0000 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:06.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.095+0000 7fef5c08d640 1 Processor -- start 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.095+0000 7fef5c08d640 1 -- start start 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.095+0000 7fef5c08d640 10 monclient: init 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 10 monclient(hunting): picked mon.noname-a con 0x7fef540a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 10 monclient(hunting): picked mon.noname-b con 0x7fef5413f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 10 monclient(hunting): picked mon.noname-c con 0x7fef541430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef54142520 con 0x7fef540a98f0 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef540b28a0 con 0x7fef541430c0 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef540b2a30 con 0x7fef5413f710 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.096+0000 7fef5c08d640 10 monclient(hunting): authenticate will time out at 4953.223331s 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.097+0000 7fef5a62a640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fef541430c0 0x7fef5414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:46054/0 (socket says 172.21.15.18:46054) 2024-03-25T19:42:06.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.097+0000 7fef5a62a640 1 -- 172.21.15.18:0/2151491804 learned_addr learned my addr 172.21.15.18:0/2151491804 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:06.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.190+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589313366 0 0) 0x7fef54142520 con 0x7fef540a98f0 2024-03-25T19:42:06.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.191+0000 7fef58e27640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.191+0000 7fef58e27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.191+0000 7fef58e27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.191+0000 7fef58e27640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:42:06.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.191+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef40003520 con 0x7fef540a98f0 2024-03-25T19:42:06.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.201+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008251588 0 0) 0x7fef540b2a30 con 0x7fef5413f710 2024-03-25T19:42:06.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.201+0000 7fef58e27640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.201+0000 7fef58e27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.201+0000 7fef58e27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.201+0000 7fef58e27640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:42:06.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.201+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef54142520 con 0x7fef5413f710 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631706161 0 0) 0x7fef40003520 con 0x7fef540a98f0 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 >> v1:172.21.15.18:6790/0 conn(0x7fef5413f710 legacy=0x7fef541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 >> v1:172.21.15.53:6789/0 conn(0x7fef541430c0 legacy=0x7fef5414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 10 monclient: _finish_hunting 0 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 1 monclient: found mon.noname-a 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef540b44c0 con 0x7fef540a98f0 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 10 monclient: _finish_auth 0 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 10 monclient: _check_auth_tickets 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fef50001690 con 0x7fef540a98f0 2024-03-25T19:42:06.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef5c08d640 5 monclient: authenticate success, global_id 4278 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef5c08d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef58e27640 10 monclient: dump: 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:06.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.211+0000 7fef5c08d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:06.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.214+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef500017f0 con 0x7fef540a98f0 2024-03-25T19:42:06.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.214+0000 7fef58e27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:06.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.214+0000 7fef58e27640 1 -- 172.21.15.18:0/2151491804 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fef50001c00 con 0x7fef540a98f0 2024-03-25T19:42:06.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.214+0000 7fef58e27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.214+0000 7fef58e27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:06.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.214+0000 7fef58e27640 10 monclient: dump: 2024-03-25T19:42:06.214 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:06.214 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.214+0000 7fef5c08d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:06.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.214+0000 7fef5c08d640 4 set_mon_vals no callback set 2024-03-25T19:42:06.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.215+0000 7fef5c08d640 10 monclient: shutdown 2024-03-25T19:42:06.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.215+0000 7fef5c08d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:06.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.215+0000 7fef5c08d640 1 -- 172.21.15.18:0/2151491804 >> v1:172.21.15.18:6789/0 conn(0x7fef540a98f0 legacy=0x7fef540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.215+0000 7fef5c08d640 1 -- 172.21.15.18:0/2151491804 shutdown_connections 2024-03-25T19:42:06.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.215+0000 7fef5c08d640 1 -- 172.21.15.18:0/2151491804 >> 172.21.15.18:0/2151491804 conn(0x7fef540b7ea0 msgr2=0x7fef540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:06.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.216+0000 7fef5c08d640 1 -- 172.21.15.18:0/2151491804 shutdown_connections 2024-03-25T19:42:06.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.216+0000 7fef5c08d640 1 -- 172.21.15.18:0/2151491804 wait complete. 2024-03-25T19:42:06.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.216+0000 7fef5c08d640 10 monclient: build_initial_monmap 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.216+0000 7fef5c08d640 10 monclient: monmap: 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 librados: starting msgr at 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 librados: starting objecter 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 Processor -- start 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 -- start start 2024-03-25T19:42:06.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 librados: setting wanted keys 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 librados: calling monclient init 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 10 monclient: init 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 10 monclient(hunting): picked mon.noname-c con 0x7fef540a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 10 monclient(hunting): picked mon.noname-a con 0x7fef5413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 10 monclient(hunting): picked mon.noname-b con 0x7fef541430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef54138970 con 0x7fef5413f710 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef541d7810 con 0x7fef541430c0 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef54139990 con 0x7fef540a98f0 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:06.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.217+0000 7fef5c08d640 10 monclient(hunting): authenticate will time out at 4953.344647s 2024-03-25T19:42:06.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.241+0000 7fef59628640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fef5413f710 0x7fef541d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:51662/0 (socket says 172.21.15.18:51662) 2024-03-25T19:42:06.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.241+0000 7fef59628640 1 -- 172.21.15.18:0/1013947351 learned_addr learned my addr 172.21.15.18:0/1013947351 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:06.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.242+0000 7fef4a7fc640 10 client.?.objecter ms_handle_connect 0x7fef5413f710 2024-03-25T19:42:06.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.242+0000 7fef4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.248+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362140003 0 0) 0x7fef54138970 con 0x7fef5413f710 2024-03-25T19:42:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.248+0000 7fef4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.248+0000 7fef4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.248+0000 7fef4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.248+0000 7fef4a7fc640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:42:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.248+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef38004eb0 con 0x7fef5413f710 2024-03-25T19:42:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.253+0000 7fef4a7fc640 10 client.?.objecter ms_handle_connect 0x7fef540a98f0 2024-03-25T19:42:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.253+0000 7fef4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.262+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45617294 0 0) 0x7fef54139990 con 0x7fef540a98f0 2024-03-25T19:42:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.262+0000 7fef4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.262+0000 7fef4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.262+0000 7fef4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.262+0000 7fef4a7fc640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:42:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.262+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef54138970 con 0x7fef540a98f0 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.264+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 971572536 0 0) 0x7fef38004eb0 con 0x7fef5413f710 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.264+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef54139990 con 0x7fef5413f710 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.264+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fef44001a10 con 0x7fef5413f710 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.264+0000 7fef4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.264+0000 7fef4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.264+0000 7fef4a7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:06.264 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:06.265 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:06.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.264+0000 7fef4a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:06.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.268+0000 7fef4a7fc640 10 client.?.objecter ms_handle_connect 0x7fef541430c0 2024-03-25T19:42:06.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.268+0000 7fef4a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.274+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3897795469 0 0) 0x7fef54139990 con 0x7fef5413f710 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.274+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 >> v1:172.21.15.18:6790/0 conn(0x7fef540a98f0 legacy=0x7fef54138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 >> v1:172.21.15.53:6789/0 conn(0x7fef541430c0 legacy=0x7fef541d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 1 monclient: found mon.a 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef541d8440 con 0x7fef5413f710 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 5 monclient: authenticate success, global_id 4281 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 10 monclient: _renew_subs 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef541d41e0 con 0x7fef5413f710 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:06.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 10 monclient: _renew_subs 2024-03-25T19:42:06.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:06.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef54138d80 con 0x7fef5413f710 2024-03-25T19:42:06.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.275+0000 7fef5c08d640 1 librados: init done 2024-03-25T19:42:06.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.276+0000 7fef5c08d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.276+0000 7fef5c08d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.276+0000 7fef5c08d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:06.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.276+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef541d79a0 con 0x7fef5413f710 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.592+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef44001c50 con 0x7fef5413f710 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.592+0000 7fef4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.592+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fef44003a60 con 0x7fef5413f710 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.592+0000 7fef4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.592+0000 7fef4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.592+0000 7fef4a7fc640 10 monclient: dump: 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:06.592 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.593 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:06.593 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:06.593 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:06.593 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:06.593 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:06.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.592+0000 7fef4bfff640 4 set_mon_vals no callback set 2024-03-25T19:42:06.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.593+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fef44003c80 con 0x7fef5413f710 2024-03-25T19:42:06.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.594+0000 7fef59e29640 10 monclient: get_auth_request con 0x7fef3806dd10 auth_method 0 2024-03-25T19:42:06.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.594+0000 7fef4a7fc640 10 client.4281.objecter ms_handle_connect 0x7fef3806dd10 2024-03-25T19:42:06.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.599+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fef44054820 con 0x7fef5413f710 2024-03-25T19:42:06.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.599+0000 7fef4a7fc640 10 client.4281.objecter ms_dispatch 0x7fef54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:06.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.599+0000 7fef4a7fc640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:06.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.599+0000 7fef4a7fc640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:06.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.599+0000 7fef4a7fc640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-25T19:42:06.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.599+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef44050330 con 0x7fef5413f710 2024-03-25T19:42:06.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.599+0000 7fef4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.599+0000 7fef4a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:06.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.861+0000 7fef5c08d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.861+0000 7fef5c08d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.861+0000 7fef5c08d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:06.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.861+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef540627b0 con 0x7fef5413f710 2024-03-25T19:42:06.915 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:34.446 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:06.946 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:34.477 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:06.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.957+0000 7fef4a7fc640 1 -- 172.21.15.18:0/1013947351 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fef44052260 con 0x7fef5413f710 2024-03-25T19:42:06.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.957+0000 7fef4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.957+0000 7fef4a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:06.957 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:42:06.957 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:06.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.959+0000 7fef5c08d640 10 librados: watch_flush enter 2024-03-25T19:42:06.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.959+0000 7fef5c08d640 10 librados: watch_flush exit 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.959+0000 7fef5c08d640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.959+0000 7fef5c08d640 10 client.4281.objecter successfully canceled tick 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 >> v1:172.21.15.18:6800/1520000573 conn(0x7fef3806dd10 legacy=0x7fef380701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 10 monclient: shutdown 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 >> v1:172.21.15.18:6789/0 conn(0x7fef5413f710 legacy=0x7fef541d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 shutdown_connections 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 >> 172.21.15.18:0/1013947351 conn(0x7fef540b7ea0 msgr2=0x7fef5413c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 shutdown_connections 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 1 -- 172.21.15.18:0/1013947351 wait complete. 2024-03-25T19:42:06.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:06.960+0000 7fef5c08d640 1 librados: shutdown 2024-03-25T19:42:06.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:06.991 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:34.488 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:06.996 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:34.527 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:07.001 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:34.497 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:07.010 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:34.541 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:07.025 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:34.556 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:07.052 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:34.549 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:07.079 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:34.610 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:13.781 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.895+0000 7fe31669d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.895+0000 7fe31669d640 10 monclient: build_initial_monmap 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.895+0000 7fe31669d640 10 monclient: monmap: 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:42:13.896189+0000 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:42:13.896189+0000 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:42:13.895 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:13.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.896+0000 7fe31669d640 1 Processor -- start 2024-03-25T19:42:13.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.896+0000 7fe31669d640 1 -- start start 2024-03-25T19:42:13.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.896+0000 7fe31669d640 10 monclient: init 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3100a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 10 monclient(hunting): picked mon.noname-c con 0x7fe31013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3101430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe310142520 con 0x7fe3100a98f0 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3100b28a0 con 0x7fe31013f710 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3100b2a30 con 0x7fe3101430c0 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:13.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.897+0000 7fe31669d640 10 monclient(hunting): authenticate will time out at 4961.024209s 2024-03-25T19:42:13.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.911+0000 7fe30ffff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe3100a98f0 0x7fe3100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:56096/0 (socket says 172.21.15.18:56096) 2024-03-25T19:42:13.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.911+0000 7fe30ffff640 1 -- 172.21.15.18:0/2154815087 learned_addr learned my addr 172.21.15.18:0/2154815087 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.937+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576517860 0 0) 0x7fe310142520 con 0x7fe3100a98f0 2024-03-25T19:42:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.937+0000 7fe30f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.937+0000 7fe30f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.937+0000 7fe30f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.937+0000 7fe30f7fe640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:42:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.937+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc003520 con 0x7fe3100a98f0 2024-03-25T19:42:13.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2229157443 0 0) 0x7fe2fc003520 con 0x7fe3100a98f0 2024-03-25T19:42:13.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 >> v1:172.21.15.18:6790/0 conn(0x7fe3101430c0 legacy=0x7fe31014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 >> v1:172.21.15.53:6789/0 conn(0x7fe31013f710 legacy=0x7fe3101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:13.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 1 monclient: found mon.noname-a 2024-03-25T19:42:13.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100b44c0 con 0x7fe3100a98f0 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe300001690 con 0x7fe3100a98f0 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe31669d640 5 monclient: authenticate success, global_id 4284 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe31669d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.949+0000 7fe30f7fe640 10 monclient: dump: 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:13.950 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:13.951 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:13.951 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:13.951 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:13.951 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:13.951 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:13.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:13.950+0000 7fe31669d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:14.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.008+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3000017f0 con 0x7fe3100a98f0 2024-03-25T19:42:14.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.008+0000 7fe30f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:14.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.008+0000 7fe30f7fe640 1 -- 172.21.15.18:0/2154815087 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe300001c00 con 0x7fe3100a98f0 2024-03-25T19:42:14.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.008+0000 7fe30f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.008+0000 7fe30f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:14.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.008+0000 7fe30f7fe640 10 monclient: dump: 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.008+0000 7fe31669d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:14.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.008+0000 7fe31669d640 4 set_mon_vals no callback set 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.009+0000 7fe31669d640 10 monclient: shutdown 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.009+0000 7fe31669d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.009+0000 7fe31669d640 1 -- 172.21.15.18:0/2154815087 >> v1:172.21.15.18:6789/0 conn(0x7fe3100a98f0 legacy=0x7fe3100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.009+0000 7fe31669d640 1 -- 172.21.15.18:0/2154815087 shutdown_connections 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.009+0000 7fe31669d640 1 -- 172.21.15.18:0/2154815087 >> 172.21.15.18:0/2154815087 conn(0x7fe3100b7ea0 msgr2=0x7fe3100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.009+0000 7fe31669d640 1 -- 172.21.15.18:0/2154815087 shutdown_connections 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.009+0000 7fe31669d640 1 -- 172.21.15.18:0/2154815087 wait complete. 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 10 monclient: build_initial_monmap 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 10 monclient: monmap: 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:42:14.012 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 1 librados: starting msgr at 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 1 librados: starting objecter 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 1 Processor -- start 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 1 -- start start 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 1 librados: setting wanted keys 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 1 librados: calling monclient init 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.010+0000 7fe31669d640 10 monclient: init 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:14.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 10 monclient(hunting): picked mon.noname-c con 0x7fe3100a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 10 monclient(hunting): picked mon.noname-b con 0x7fe31013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3101430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe310138c70 con 0x7fe3101430c0 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3101367d0 con 0x7fe31013f710 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe310136960 con 0x7fe3100a98f0 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.011+0000 7fe31669d640 10 monclient(hunting): authenticate will time out at 4961.138242s 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.012+0000 7fe307fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe31013f710 0x7fe310132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:50718/0 (socket says 172.21.15.18:50718) 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.012+0000 7fe307fff640 1 -- 172.21.15.18:0/1132368850 learned_addr learned my addr 172.21.15.18:0/1132368850 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.012+0000 7fe30d7fa640 10 client.?.objecter ms_handle_connect 0x7fe31013f710 2024-03-25T19:42:14.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.012+0000 7fe30d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:14.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.024+0000 7fe30d7fa640 10 client.?.objecter ms_handle_connect 0x7fe3101430c0 2024-03-25T19:42:14.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.024+0000 7fe30d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:14.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.048+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278885593 0 0) 0x7fe310138c70 con 0x7fe3101430c0 2024-03-25T19:42:14.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.048+0000 7fe30d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.048+0000 7fe30d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.048+0000 7fe30d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.048+0000 7fe30d7fa640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:42:14.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.048+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f0004eb0 con 0x7fe3101430c0 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.069+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1027791298 0 0) 0x7fe2f0004eb0 con 0x7fe3101430c0 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.069+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe310138c70 con 0x7fe3101430c0 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.069+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe308001b90 con 0x7fe3101430c0 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.069+0000 7fe30d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.069+0000 7fe30d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.069+0000 7fe30d7fa640 10 monclient(hunting): dump: 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:14.069 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.070 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:14.070 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:14.070 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:14.070 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:14.070 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:14.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.069+0000 7fe30d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.080+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 211640340 0 0) 0x7fe310138c70 con 0x7fe3101430c0 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 >> v1:172.21.15.18:6790/0 conn(0x7fe3100a98f0 legacy=0x7fe310137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 >> v1:172.21.15.53:6789/0 conn(0x7fe31013f710 legacy=0x7fe310132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 1 monclient: found mon.a 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe310139630 con 0x7fe3101430c0 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe30d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:14.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 5 monclient: authenticate success, global_id 4287 2024-03-25T19:42:14.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 10 monclient: _renew_subs 2024-03-25T19:42:14.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:14.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe310139880 con 0x7fe3101430c0 2024-03-25T19:42:14.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:14.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 10 monclient: _renew_subs 2024-03-25T19:42:14.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:14.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3101337f0 con 0x7fe3101430c0 2024-03-25T19:42:14.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.081+0000 7fe31669d640 1 librados: init done 2024-03-25T19:42:14.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.082+0000 7fe31669d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.082+0000 7fe31669d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.082+0000 7fe31669d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:14.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.082+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe310133aa0 con 0x7fe3101430c0 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.351+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe308003990 con 0x7fe3101430c0 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.351+0000 7fe30d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.351+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe308002100 con 0x7fe3101430c0 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.351+0000 7fe30d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.351+0000 7fe30d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.351+0000 7fe30d7fa640 10 monclient: dump: 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:14.351 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.352 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:14.352 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:14.352 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:14.352 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:14.352 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:14.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.351+0000 7fe30effd640 4 set_mon_vals no callback set 2024-03-25T19:42:14.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.352+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fe308003e60 con 0x7fe3101430c0 2024-03-25T19:42:14.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.352+0000 7fe30ffff640 10 monclient: get_auth_request con 0x7fe2f006dc70 auth_method 0 2024-03-25T19:42:14.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.352+0000 7fe30d7fa640 10 client.4287.objecter ms_handle_connect 0x7fe2f006dc70 2024-03-25T19:42:14.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.357+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fe3080538b0 con 0x7fe3101430c0 2024-03-25T19:42:14.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.357+0000 7fe30d7fa640 10 client.4287.objecter ms_dispatch 0x7fe310000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:14.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.357+0000 7fe30d7fa640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:14.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.357+0000 7fe30d7fa640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:14.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.357+0000 7fe30d7fa640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-25T19:42:14.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.357+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe30804f3c0 con 0x7fe3101430c0 2024-03-25T19:42:14.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.357+0000 7fe30d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.357+0000 7fe30d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:14.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.594+0000 7fe31669d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.594+0000 7fe31669d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.594+0000 7fe31669d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:14.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.594+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3100627b0 con 0x7fe3101430c0 2024-03-25T19:42:14.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.636+0000 7fe30d7fa640 1 -- 172.21.15.18:0/1132368850 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fe3080512f0 con 0x7fe3101430c0 2024-03-25T19:42:14.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.636+0000 7fe30d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.636+0000 7fe30d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:14.636 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:42:14.636 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 10 librados: watch_flush enter 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 10 librados: watch_flush exit 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 10 client.4287.objecter successfully canceled tick 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 >> v1:172.21.15.18:6800/1520000573 conn(0x7fe2f006dc70 legacy=0x7fe2f0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 10 monclient: shutdown 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 >> v1:172.21.15.18:6789/0 conn(0x7fe3101430c0 legacy=0x7fe3101e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 shutdown_connections 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 >> 172.21.15.18:0/1132368850 conn(0x7fe3100b7ea0 msgr2=0x7fe310074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:14.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 shutdown_connections 2024-03-25T19:42:14.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 1 -- 172.21.15.18:0/1132368850 wait complete. 2024-03-25T19:42:14.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:14.639+0000 7fe31669d640 1 librados: shutdown 2024-03-25T19:42:14.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:21.457 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:21.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.573+0000 7fca698ff640 10 monclient: get_monmap_and_config 2024-03-25T19:42:21.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.573+0000 7fca698ff640 10 monclient: build_initial_monmap 2024-03-25T19:42:21.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.573+0000 7fca698ff640 10 monclient: monmap: 2024-03-25T19:42:21.573 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:21.573 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:21.573 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:42:21.574642+0000 2024-03-25T19:42:21.573 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:42:21.574642+0000 2024-03-25T19:42:21.573 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:21.574 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:21.574 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:21.574 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:42:21.574 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:42:21.574 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.574+0000 7fca698ff640 1 Processor -- start 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.574+0000 7fca698ff640 1 -- start start 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.574+0000 7fca698ff640 10 monclient: init 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 10 monclient(hunting): picked mon.noname-a con 0x7fca640a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 10 monclient(hunting): picked mon.noname-b con 0x7fca6413f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 10 monclient(hunting): picked mon.noname-c con 0x7fca641430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:21.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca64142520 con 0x7fca640a98f0 2024-03-25T19:42:21.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640b28a0 con 0x7fca641430c0 2024-03-25T19:42:21.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640b2a30 con 0x7fca6413f710 2024-03-25T19:42:21.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:21.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.575+0000 7fca698ff640 10 monclient(hunting): authenticate will time out at 4968.702148s 2024-03-25T19:42:21.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.580+0000 7fca637fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fca641430c0 0x7fca6414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:43992/0 (socket says 172.21.15.18:43992) 2024-03-25T19:42:21.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.580+0000 7fca637fe640 1 -- 172.21.15.18:0/2063197383 learned_addr learned my addr 172.21.15.18:0/2063197383 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:21.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.589+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890920349 0 0) 0x7fca640b28a0 con 0x7fca641430c0 2024-03-25T19:42:21.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.589+0000 7fca61ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.589+0000 7fca61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.589+0000 7fca61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.589+0000 7fca61ffb640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:42:21.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.589+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca4c003520 con 0x7fca641430c0 2024-03-25T19:42:21.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.598+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934788443 0 0) 0x7fca640b2a30 con 0x7fca6413f710 2024-03-25T19:42:21.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.598+0000 7fca61ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.598+0000 7fca61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.598+0000 7fca61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.598+0000 7fca61ffb640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:42:21.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.598+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca640b28a0 con 0x7fca6413f710 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2091486987 0 0) 0x7fca4c003520 con 0x7fca641430c0 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 >> v1:172.21.15.18:6790/0 conn(0x7fca6413f710 legacy=0x7fca641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 >> v1:172.21.15.18:6789/0 conn(0x7fca640a98f0 legacy=0x7fca640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 1 monclient: found mon.noname-c 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640b44c0 con 0x7fca641430c0 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:21.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fca5c001a10 con 0x7fca641430c0 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca61ffb640 10 monclient: dump: 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:21.611 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:21.612 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:21.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca698ff640 5 monclient: authenticate success, global_id 4264 2024-03-25T19:42:21.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.610+0000 7fca698ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:21.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca5c001b70 con 0x7fca641430c0 2024-03-25T19:42:21.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:21.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca61ffb640 1 -- 172.21.15.18:0/2063197383 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fca5c001f80 con 0x7fca641430c0 2024-03-25T19:42:21.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca698ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:21.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca698ff640 4 set_mon_vals no callback set 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca61ffb640 10 monclient: dump: 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:21.637 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:21.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca698ff640 10 monclient: shutdown 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca698ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.637+0000 7fca698ff640 1 -- 172.21.15.18:0/2063197383 >> v1:172.21.15.53:6789/0 conn(0x7fca641430c0 legacy=0x7fca6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.638+0000 7fca698ff640 1 -- 172.21.15.18:0/2063197383 shutdown_connections 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.638+0000 7fca698ff640 1 -- 172.21.15.18:0/2063197383 >> 172.21.15.18:0/2063197383 conn(0x7fca640b7ea0 msgr2=0x7fca640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.638+0000 7fca698ff640 1 -- 172.21.15.18:0/2063197383 shutdown_connections 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.638+0000 7fca698ff640 1 -- 172.21.15.18:0/2063197383 wait complete. 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.638+0000 7fca698ff640 10 monclient: build_initial_monmap 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.638+0000 7fca698ff640 10 monclient: monmap: 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:21.640 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.639+0000 7fca698ff640 1 librados: starting msgr at 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.639+0000 7fca698ff640 1 librados: starting objecter 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.639+0000 7fca698ff640 1 Processor -- start 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.639+0000 7fca698ff640 1 -- start start 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.639+0000 7fca698ff640 1 librados: setting wanted keys 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.639+0000 7fca698ff640 1 librados: calling monclient init 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.639+0000 7fca698ff640 10 monclient: init 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 10 monclient(hunting): picked mon.noname-b con 0x7fca640a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:21.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 10 monclient(hunting): picked mon.noname-c con 0x7fca6413f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:21.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 10 monclient(hunting): picked mon.noname-a con 0x7fca641430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:21.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca641369d0 con 0x7fca641430c0 2024-03-25T19:42:21.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca64133a60 con 0x7fca640a98f0 2024-03-25T19:42:21.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640ab9e0 con 0x7fca6413f710 2024-03-25T19:42:21.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:21.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.640+0000 7fca698ff640 10 monclient(hunting): authenticate will time out at 4968.767066s 2024-03-25T19:42:21.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.650+0000 7fca627fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fca6413f710 0x7fca641362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54834/0 (socket says 172.21.15.18:54834) 2024-03-25T19:42:21.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.650+0000 7fca627fc640 1 -- 172.21.15.18:0/2904037630 learned_addr learned my addr 172.21.15.18:0/2904037630 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:21.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.652+0000 7fca4bfff640 10 client.?.objecter ms_handle_connect 0x7fca6413f710 2024-03-25T19:42:21.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.652+0000 7fca4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.656+0000 7fca4bfff640 10 client.?.objecter ms_handle_connect 0x7fca640a98f0 2024-03-25T19:42:21.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.656+0000 7fca4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.657+0000 7fca4bfff640 10 client.?.objecter ms_handle_connect 0x7fca641430c0 2024-03-25T19:42:21.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.657+0000 7fca4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.665+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632881848 0 0) 0x7fca640ab9e0 con 0x7fca6413f710 2024-03-25T19:42:21.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.665+0000 7fca4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.665+0000 7fca4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.665+0000 7fca4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.665+0000 7fca4bfff640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:42:21.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.665+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca50004eb0 con 0x7fca6413f710 2024-03-25T19:42:21.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.669+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727404347 0 0) 0x7fca64133a60 con 0x7fca640a98f0 2024-03-25T19:42:21.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.669+0000 7fca4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.669+0000 7fca4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.669+0000 7fca4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.669+0000 7fca4bfff640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:42:21.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.669+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca640ab9e0 con 0x7fca640a98f0 2024-03-25T19:42:21.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.674+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053615344 0 0) 0x7fca641369d0 con 0x7fca641430c0 2024-03-25T19:42:21.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.674+0000 7fca4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.674+0000 7fca4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.674+0000 7fca4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.674+0000 7fca4bfff640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:42:21.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.674+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64133a60 con 0x7fca641430c0 2024-03-25T19:42:21.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.678+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1856368800 0 0) 0x7fca50004eb0 con 0x7fca6413f710 2024-03-25T19:42:21.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.678+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca641369d0 con 0x7fca6413f710 2024-03-25T19:42:21.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.678+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fca58001a10 con 0x7fca6413f710 2024-03-25T19:42:21.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.678+0000 7fca4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.678+0000 7fca4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:21.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.678+0000 7fca4bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:21.678 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:21.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.678+0000 7fca4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:21.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.683+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2785480358 0 0) 0x7fca64133a60 con 0x7fca641430c0 2024-03-25T19:42:21.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.683+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca50004eb0 con 0x7fca641430c0 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.683+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fca5c001e80 con 0x7fca641430c0 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.683+0000 7fca4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.683+0000 7fca4bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.683+0000 7fca4bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:21.683 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:21.684 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:21.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.684+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 71773231 0 0) 0x7fca640ab9e0 con 0x7fca640a98f0 2024-03-25T19:42:21.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.684+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca64133a60 con 0x7fca640a98f0 2024-03-25T19:42:21.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.684+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fca540018b0 con 0x7fca640a98f0 2024-03-25T19:42:21.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.684+0000 7fca4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.684+0000 7fca4bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.684+0000 7fca4bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:21.685 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 538828891 0 0) 0x7fca641369d0 con 0x7fca6413f710 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 >> v1:172.21.15.53:6789/0 conn(0x7fca640a98f0 legacy=0x7fca64132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 >> v1:172.21.15.18:6789/0 conn(0x7fca641430c0 legacy=0x7fca64133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 1 monclient: found mon.c 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640abcc0 con 0x7fca6413f710 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 10 monclient: _finish_auth 0 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:21.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 5 monclient: authenticate success, global_id 4271 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 10 monclient: _renew_subs 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca640abfa0 con 0x7fca6413f710 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 10 monclient: _renew_subs 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca64137e40 con 0x7fca6413f710 2024-03-25T19:42:21.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.695+0000 7fca698ff640 1 librados: init done 2024-03-25T19:42:21.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.696+0000 7fca698ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:21.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.696+0000 7fca698ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:21.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.696+0000 7fca698ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:42:21.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:21.696+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca640627b0 con 0x7fca6413f710 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.013+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca58001c50 con 0x7fca6413f710 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.013+0000 7fca4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.013+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fca58003a60 con 0x7fca6413f710 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.013+0000 7fca4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.013+0000 7fca4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.013+0000 7fca4bfff640 10 monclient: dump: 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:22.013 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:22.014 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:22.014 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:22.014 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:22.014 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:22.014 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:22.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.013+0000 7fca617fa640 4 set_mon_vals no callback set 2024-03-25T19:42:22.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.014+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fca58003c80 con 0x7fca6413f710 2024-03-25T19:42:22.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.015+0000 7fca62ffd640 10 monclient: get_auth_request con 0x7fca5006e0c0 auth_method 0 2024-03-25T19:42:22.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.015+0000 7fca4bfff640 10 client.4271.objecter ms_handle_connect 0x7fca5006e0c0 2024-03-25T19:42:22.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.022+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fca580548b0 con 0x7fca6413f710 2024-03-25T19:42:22.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.022+0000 7fca4bfff640 10 client.4271.objecter ms_dispatch 0x7fca64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:22.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.022+0000 7fca4bfff640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:22.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.022+0000 7fca4bfff640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:22.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.022+0000 7fca4bfff640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-25T19:42:22.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.022+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca580503c0 con 0x7fca6413f710 2024-03-25T19:42:22.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.022+0000 7fca4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:22.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.022+0000 7fca4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:22.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.250+0000 7fca698ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.250+0000 7fca698ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.250+0000 7fca698ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:42:22.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.250+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca64139000 con 0x7fca6413f710 2024-03-25T19:42:22.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.348+0000 7fca4bfff640 1 -- 172.21.15.18:0/2904037630 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fca580522f0 con 0x7fca6413f710 2024-03-25T19:42:22.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.348+0000 7fca4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.348+0000 7fca4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:22.348 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:42:22.348 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:22.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 10 librados: watch_flush enter 2024-03-25T19:42:22.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 10 librados: watch_flush exit 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 10 client.4271.objecter successfully canceled tick 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 >> v1:172.21.15.18:6800/1520000573 conn(0x7fca5006e0c0 legacy=0x7fca50070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 10 monclient: shutdown 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 >> v1:172.21.15.18:6790/0 conn(0x7fca6413f710 legacy=0x7fca641362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 shutdown_connections 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.351+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 >> 172.21.15.18:0/2904037630 conn(0x7fca640b7ea0 msgr2=0x7fca64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.352+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 shutdown_connections 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.352+0000 7fca698ff640 1 -- 172.21.15.18:0/2904037630 wait complete. 2024-03-25T19:42:22.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:22.352+0000 7fca698ff640 1 librados: shutdown 2024-03-25T19:42:22.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:26.987 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:54.518 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:26.996 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:54.526 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:27.019 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:54.550 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:27.040 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:54.536 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:27.043 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:54.539 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:27.054 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:54.585 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:27.060 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:54.591 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:27.083 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:01:54.579 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:27.101 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:54.632 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:29.168 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.280+0000 7f3552c93640 10 monclient: get_monmap_and_config 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.280+0000 7f3552c93640 10 monclient: build_initial_monmap 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.280+0000 7f3552c93640 10 monclient: monmap: 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:42:29.282430+0000 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:42:29.282430+0000 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:29.281 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:29.282 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:42:29.282 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:42:29.282 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:29.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.281+0000 7f3552c93640 1 Processor -- start 2024-03-25T19:42:29.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.281+0000 7f3552c93640 1 -- start start 2024-03-25T19:42:29.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.281+0000 7f3552c93640 10 monclient: init 2024-03-25T19:42:29.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:29.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:29.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 10 monclient(hunting): picked mon.noname-a con 0x7f354c06ab80 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:29.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 10 monclient(hunting): picked mon.noname-c con 0x7f354c06af60 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:29.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 10 monclient(hunting): picked mon.noname-b con 0x7f354c0b00b0 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:29.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c0b0500 con 0x7f354c06ab80 2024-03-25T19:42:29.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c13ffa0 con 0x7f354c06af60 2024-03-25T19:42:29.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c0af520 con 0x7f354c0b00b0 2024-03-25T19:42:29.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:29.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.282+0000 7f3552c93640 10 monclient(hunting): authenticate will time out at 4976.410065s 2024-03-25T19:42:29.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.288+0000 7f354bfff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f354c06af60 0x7f354c13f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:59544/0 (socket says 172.21.15.18:59544) 2024-03-25T19:42:29.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.288+0000 7f354bfff640 1 -- 172.21.15.18:0/1445090476 learned_addr learned my addr 172.21.15.18:0/1445090476 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:29.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.301+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15926239 0 0) 0x7f354c13ffa0 con 0x7f354c06af60 2024-03-25T19:42:29.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.301+0000 7f354b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.301+0000 7f354b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.301+0000 7f354b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.301+0000 7f354b7fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:42:29.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.301+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c003520 con 0x7f354c06af60 2024-03-25T19:42:29.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.307+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662348606 0 0) 0x7f354c0b0500 con 0x7f354c06ab80 2024-03-25T19:42:29.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.308+0000 7f354b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.308+0000 7f354b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.308+0000 7f354b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.308+0000 7f354b7fe640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:42:29.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.308+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f354c13ffa0 con 0x7f354c06ab80 2024-03-25T19:42:29.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941700389 0 0) 0x7f353c003520 con 0x7f354c06af60 2024-03-25T19:42:29.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 >> v1:172.21.15.18:6790/0 conn(0x7f354c0b00b0 legacy=0x7f354c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 >> v1:172.21.15.18:6789/0 conn(0x7f354c06ab80 legacy=0x7f354c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:29.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:42:29.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:29.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:42:29.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c072890 con 0x7f354c06af60 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3538001690 con 0x7f354c06af60 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f354b7fe640 10 monclient: dump: 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:29.312 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:29.313 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:29.313 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:29.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f3552c93640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:42:29.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.310+0000 7f3552c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.335+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35380017f0 con 0x7f354c06af60 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.335+0000 7f354b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.335+0000 7f354b7fe640 1 -- 172.21.15.18:0/1445090476 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3538001c00 con 0x7f354c06af60 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.335+0000 7f3552c93640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.335+0000 7f3552c93640 4 set_mon_vals no callback set 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.335+0000 7f354b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.335+0000 7f354b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.335+0000 7f354b7fe640 10 monclient: dump: 2024-03-25T19:42:29.336 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:29.337 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.336+0000 7f3552c93640 10 monclient: shutdown 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.336+0000 7f3552c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.336+0000 7f3552c93640 1 -- 172.21.15.18:0/1445090476 >> v1:172.21.15.53:6789/0 conn(0x7f354c06af60 legacy=0x7f354c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.336+0000 7f3552c93640 1 -- 172.21.15.18:0/1445090476 shutdown_connections 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.336+0000 7f3552c93640 1 -- 172.21.15.18:0/1445090476 >> 172.21.15.18:0/1445090476 conn(0x7f354c0b4a80 msgr2=0x7f354c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.336+0000 7f3552c93640 1 -- 172.21.15.18:0/1445090476 shutdown_connections 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.336+0000 7f3552c93640 1 -- 172.21.15.18:0/1445090476 wait complete. 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.337+0000 7f3552c93640 10 monclient: build_initial_monmap 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.337+0000 7f3552c93640 10 monclient: monmap: 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:42:29.339 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.338+0000 7f3552c93640 1 librados: starting msgr at 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.338+0000 7f3552c93640 1 librados: starting objecter 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.338+0000 7f3552c93640 1 Processor -- start 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.338+0000 7f3552c93640 1 -- start start 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.338+0000 7f3552c93640 1 librados: setting wanted keys 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.338+0000 7f3552c93640 1 librados: calling monclient init 2024-03-25T19:42:29.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.338+0000 7f3552c93640 10 monclient: init 2024-03-25T19:42:29.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:29.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:29.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 10 monclient(hunting): picked mon.noname-c con 0x7f354c06ab80 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:29.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 10 monclient(hunting): picked mon.noname-a con 0x7f354c0b00b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:29.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 10 monclient(hunting): picked mon.noname-b con 0x7f354c146580 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:29.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c11d4f0 con 0x7f354c0b00b0 2024-03-25T19:42:29.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c13b340 con 0x7f354c146580 2024-03-25T19:42:29.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c11e670 con 0x7f354c06ab80 2024-03-25T19:42:29.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:29.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.339+0000 7f3552c93640 10 monclient(hunting): authenticate will time out at 4976.467209s 2024-03-25T19:42:29.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.346+0000 7f3551230640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f354c146580 0x7f354c1e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:59558/0 (socket says 172.21.15.18:59558) 2024-03-25T19:42:29.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.346+0000 7f3551230640 1 -- 172.21.15.18:0/4119960332 learned_addr learned my addr 172.21.15.18:0/4119960332 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:29.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.347+0000 7f35497fa640 10 client.?.objecter ms_handle_connect 0x7f354c146580 2024-03-25T19:42:29.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.347+0000 7f35497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.350+0000 7f35497fa640 10 client.?.objecter ms_handle_connect 0x7f354c0b00b0 2024-03-25T19:42:29.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.350+0000 7f35497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.360+0000 7f35497fa640 10 client.?.objecter ms_handle_connect 0x7f354c06ab80 2024-03-25T19:42:29.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.360+0000 7f35497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.360+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457697200 0 0) 0x7f354c11d4f0 con 0x7f354c0b00b0 2024-03-25T19:42:29.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.360+0000 7f35497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.360+0000 7f35497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.360+0000 7f35497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.360+0000 7f35497fa640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:42:29.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.360+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3530004eb0 con 0x7f354c0b00b0 2024-03-25T19:42:29.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.361+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797649422 0 0) 0x7f354c13b340 con 0x7f354c146580 2024-03-25T19:42:29.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.361+0000 7f35497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.361+0000 7f35497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.361+0000 7f35497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.361+0000 7f35497fa640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:42:29.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.361+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f354c11d4f0 con 0x7f354c146580 2024-03-25T19:42:29.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.366+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316607597 0 0) 0x7f354c11e670 con 0x7f354c06ab80 2024-03-25T19:42:29.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.366+0000 7f35497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.366+0000 7f35497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.366+0000 7f35497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.366+0000 7f35497fa640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:42:29.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.366+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f354c13b340 con 0x7f354c06ab80 2024-03-25T19:42:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.372+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 551541693 0 0) 0x7f354c13b340 con 0x7f354c06ab80 2024-03-25T19:42:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.372+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f354c11e670 con 0x7f354c06ab80 2024-03-25T19:42:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.372+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3540001930 con 0x7f354c06ab80 2024-03-25T19:42:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.372+0000 7f35497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.372+0000 7f35497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.372+0000 7f35497fa640 10 monclient(hunting): dump: 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:29.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.372+0000 7f35497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.377+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1186205314 0 0) 0x7f3530004eb0 con 0x7f354c0b00b0 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.377+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f354c13b340 con 0x7f354c0b00b0 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.377+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3538001b00 con 0x7f354c0b00b0 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.377+0000 7f35497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.377+0000 7f35497fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.377+0000 7f35497fa640 10 monclient(hunting): dump: 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:29.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.380+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3562852291 0 0) 0x7f354c11d4f0 con 0x7f354c146580 2024-03-25T19:42:29.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.380+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3530004eb0 con 0x7f354c146580 2024-03-25T19:42:29.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.380+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3544001c10 con 0x7f354c146580 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.380+0000 7f35497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.380+0000 7f35497fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.380+0000 7f35497fa640 10 monclient(hunting): dump: 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:29.381 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 235132784 0 0) 0x7f3530004eb0 con 0x7f354c146580 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 >> v1:172.21.15.18:6790/0 conn(0x7f354c06ab80 legacy=0x7f354c11cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 >> v1:172.21.15.18:6789/0 conn(0x7f354c0b00b0 legacy=0x7f354c1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 1 monclient: found mon.b 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c13b5b0 con 0x7f354c146580 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f35497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 10 monclient: _renew_subs 2024-03-25T19:42:29.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354c1db8a0 con 0x7f354c146580 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 10 monclient: _renew_subs 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f354c11c840 con 0x7f354c146580 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.394+0000 7f3552c93640 1 librados: init done 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.396+0000 7f3552c93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.396+0000 7f3552c93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.396+0000 7f3552c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:29.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.396+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354c06b530 con 0x7f354c146580 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.687+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3544003970 con 0x7f354c146580 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.687+0000 7f35497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.687+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3544002100 con 0x7f354c146580 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.687+0000 7f35497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.687+0000 7f35497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.687+0000 7f35497fa640 10 monclient: dump: 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.687+0000 7f354affd640 4 set_mon_vals no callback set 2024-03-25T19:42:29.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.688+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f3544003e00 con 0x7f354c146580 2024-03-25T19:42:29.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.689+0000 7f3550a2f640 10 monclient: get_auth_request con 0x7f353006e110 auth_method 0 2024-03-25T19:42:29.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.689+0000 7f35497fa640 10 client.4273.objecter ms_handle_connect 0x7f353006e110 2024-03-25T19:42:29.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.694+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f3544054a50 con 0x7f354c146580 2024-03-25T19:42:29.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.694+0000 7f35497fa640 10 client.4273.objecter ms_dispatch 0x7f354c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:29.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.694+0000 7f35497fa640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:29.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.694+0000 7f35497fa640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:29.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.694+0000 7f35497fa640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-25T19:42:29.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.695+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3544050560 con 0x7f354c146580 2024-03-25T19:42:29.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.695+0000 7f35497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.695+0000 7f35497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:29.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.930+0000 7f3552c93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:29.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.930+0000 7f3552c93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:29.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.930+0000 7f3552c93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:29.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:29.930+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f354c0627b0 con 0x7f354c146580 2024-03-25T19:42:30.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.002+0000 7f35497fa640 1 -- 172.21.15.18:0/4119960332 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f3544052490 con 0x7f354c146580 2024-03-25T19:42:30.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.002+0000 7f35497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:30.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.002+0000 7f35497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:30.003 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:42:30.003 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:30.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 10 librados: watch_flush enter 2024-03-25T19:42:30.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 10 librados: watch_flush exit 2024-03-25T19:42:30.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-25T19:42:30.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 10 client.4273.objecter successfully canceled tick 2024-03-25T19:42:30.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 >> v1:172.21.15.18:6800/1520000573 conn(0x7f353006e110 legacy=0x7f35300705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 10 monclient: shutdown 2024-03-25T19:42:30.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:30.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 >> v1:172.21.15.53:6789/0 conn(0x7f354c146580 legacy=0x7f354c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:30.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 shutdown_connections 2024-03-25T19:42:30.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 >> 172.21.15.18:0/4119960332 conn(0x7f354c0b4a80 msgr2=0x7f354c06f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:30.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 shutdown_connections 2024-03-25T19:42:30.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 1 -- 172.21.15.18:0/4119960332 wait complete. 2024-03-25T19:42:30.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:30.005+0000 7f3552c93640 1 librados: shutdown 2024-03-25T19:42:30.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:36.824 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:36.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.932+0000 7f19d8c57640 10 monclient: get_monmap_and_config 2024-03-25T19:42:36.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.932+0000 7f19d8c57640 10 monclient: build_initial_monmap 2024-03-25T19:42:36.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.932+0000 7f19d8c57640 10 monclient: monmap: 2024-03-25T19:42:36.933 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:36.933 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:36.934 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:42:36.934798+0000 2024-03-25T19:42:36.934 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:42:36.934798+0000 2024-03-25T19:42:36.934 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:36.934 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:36.934 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:36.934 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:42:36.934 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:42:36.934 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:36.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.933+0000 7f19d8c57640 1 Processor -- start 2024-03-25T19:42:36.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.933+0000 7f19d8c57640 1 -- start start 2024-03-25T19:42:36.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.933+0000 7f19d8c57640 10 monclient: init 2024-03-25T19:42:36.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:36.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:36.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 10 monclient(hunting): picked mon.noname-a con 0x7f19d40a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:36.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 10 monclient(hunting): picked mon.noname-c con 0x7f19d413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:36.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 10 monclient(hunting): picked mon.noname-b con 0x7f19d41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:36.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d4142520 con 0x7f19d40a98f0 2024-03-25T19:42:36.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d40b28a0 con 0x7f19d413f710 2024-03-25T19:42:36.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d40b2a30 con 0x7f19d41430c0 2024-03-25T19:42:36.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:36.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.934+0000 7f19d8c57640 10 monclient(hunting): authenticate will time out at 4984.062175s 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.962+0000 7f19d1d9b640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f19d413f710 0x7f19d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:56886/0 (socket says 172.21.15.18:56886) 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.962+0000 7f19d1d9b640 1 -- 172.21.15.18:0/2818130135 learned_addr learned my addr 172.21.15.18:0/2818130135 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.984+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613240112 0 0) 0x7f19d40b2a30 con 0x7f19d41430c0 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.984+0000 7f19d159a640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.984+0000 7f19d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.984+0000 7f19d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.984+0000 7f19d159a640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.984+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc003520 con 0x7f19d41430c0 2024-03-25T19:42:36.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.987+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007251006 0 0) 0x7f19d40b28a0 con 0x7f19d413f710 2024-03-25T19:42:36.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.987+0000 7f19d159a640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.987+0000 7f19d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.987+0000 7f19d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.987+0000 7f19d159a640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:42:36.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.987+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d40b2a30 con 0x7f19d413f710 2024-03-25T19:42:36.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1198740578 0 0) 0x7f19d40b2a30 con 0x7f19d413f710 2024-03-25T19:42:36.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 >> v1:172.21.15.18:6790/0 conn(0x7f19d41430c0 legacy=0x7f19d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 >> v1:172.21.15.18:6789/0 conn(0x7f19d40a98f0 legacy=0x7f19d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 10 monclient: _finish_hunting 0 2024-03-25T19:42:36.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 1 monclient: found mon.noname-c 2024-03-25T19:42:36.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:36.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:42:36.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d40b44c0 con 0x7f19d413f710 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 10 monclient: _finish_auth 0 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 10 monclient: _check_auth_tickets 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f19c8001670 con 0x7f19d413f710 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d159a640 10 monclient: dump: 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.995 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:36.996 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:36.996 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:36.996 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:36.996 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:36.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d8c57640 5 monclient: authenticate success, global_id 4276 2024-03-25T19:42:36.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:36.993+0000 7f19d8c57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:37.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.027+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19c80017d0 con 0x7f19d413f710 2024-03-25T19:42:37.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.027+0000 7f19d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:37.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.027+0000 7f19d159a640 1 -- 172.21.15.18:0/2818130135 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f19c8001ba0 con 0x7f19d413f710 2024-03-25T19:42:37.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.027+0000 7f19d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.027+0000 7f19d159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:37.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.027+0000 7f19d159a640 10 monclient: dump: 2024-03-25T19:42:37.028 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.027+0000 7f19d8c57640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:37.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.027+0000 7f19d8c57640 4 set_mon_vals no callback set 2024-03-25T19:42:37.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.028+0000 7f19d8c57640 10 monclient: shutdown 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.028+0000 7f19d8c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.028+0000 7f19d8c57640 1 -- 172.21.15.18:0/2818130135 >> v1:172.21.15.53:6789/0 conn(0x7f19d413f710 legacy=0x7f19d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.028+0000 7f19d8c57640 1 -- 172.21.15.18:0/2818130135 shutdown_connections 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.028+0000 7f19d8c57640 1 -- 172.21.15.18:0/2818130135 >> 172.21.15.18:0/2818130135 conn(0x7f19d40b7ea0 msgr2=0x7f19d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.029+0000 7f19d8c57640 1 -- 172.21.15.18:0/2818130135 shutdown_connections 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.029+0000 7f19d8c57640 1 -- 172.21.15.18:0/2818130135 wait complete. 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.029+0000 7f19d8c57640 10 monclient: build_initial_monmap 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.029+0000 7f19d8c57640 10 monclient: monmap: 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:42:37.032 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 1 librados: starting msgr at 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 1 librados: starting objecter 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 1 Processor -- start 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 1 -- start start 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 1 librados: setting wanted keys 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 1 librados: calling monclient init 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 10 monclient: init 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 10 monclient(hunting): picked mon.noname-a con 0x7f19d40a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 10 monclient(hunting): picked mon.noname-b con 0x7f19d413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 10 monclient(hunting): picked mon.noname-c con 0x7f19d41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.030+0000 7f19d8c57640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d41d78e0 con 0x7f19d40a98f0 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.031+0000 7f19d8c57640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d41d7500 con 0x7f19d413f710 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.031+0000 7f19d8c57640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d41d7690 con 0x7f19d41430c0 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.031+0000 7f19d8c57640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.031+0000 7f19d8c57640 10 monclient(hunting): authenticate will time out at 4984.158795s 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.032+0000 7f19d259c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f19d40a98f0 0x7f19d4134de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:40474/0 (socket says 172.21.15.18:40474) 2024-03-25T19:42:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.032+0000 7f19d259c640 1 -- 172.21.15.18:0/2861627069 learned_addr learned my addr 172.21.15.18:0/2861627069 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:37.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.038+0000 7f19c2ffd640 10 client.?.objecter ms_handle_connect 0x7f19d40a98f0 2024-03-25T19:42:37.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.038+0000 7f19c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:37.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.041+0000 7f19c2ffd640 10 client.?.objecter ms_handle_connect 0x7f19d41430c0 2024-03-25T19:42:37.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.041+0000 7f19c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:37.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.050+0000 7f19c2ffd640 10 client.?.objecter ms_handle_connect 0x7f19d413f710 2024-03-25T19:42:37.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.050+0000 7f19c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:37.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.052+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431652092 0 0) 0x7f19d41d78e0 con 0x7f19d40a98f0 2024-03-25T19:42:37.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.052+0000 7f19c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:37.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.052+0000 7f19c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:37.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.052+0000 7f19c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:37.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.052+0000 7f19c2ffd640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:42:37.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.052+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b4004eb0 con 0x7f19d40a98f0 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.061+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224299258 0 0) 0x7f19b4004eb0 con 0x7f19d40a98f0 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.061+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19d41d78e0 con 0x7f19d40a98f0 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.061+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f19c40019d0 con 0x7f19d40a98f0 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.061+0000 7f19c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.061+0000 7f19c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.061+0000 7f19c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:37.062 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.063 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:37.063 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:37.063 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:37.063 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:37.063 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:37.063 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.061+0000 7f19c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:37.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.066+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241761891 0 0) 0x7f19d41d7690 con 0x7f19d41430c0 2024-03-25T19:42:37.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.066+0000 7f19c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:37.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.066+0000 7f19c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:37.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.066+0000 7f19c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:37.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.066+0000 7f19c2ffd640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:42:37.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.066+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b4004eb0 con 0x7f19d41430c0 2024-03-25T19:42:37.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.067+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588436505 0 0) 0x7f19d41d7500 con 0x7f19d413f710 2024-03-25T19:42:37.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.067+0000 7f19c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:37.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.067+0000 7f19c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:37.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.067+0000 7f19c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:37.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.067+0000 7f19c2ffd640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:42:37.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.067+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d41d7690 con 0x7f19d413f710 2024-03-25T19:42:37.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1368791559 0 0) 0x7f19d41d78e0 con 0x7f19d40a98f0 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 >> v1:172.21.15.18:6790/0 conn(0x7f19d41430c0 legacy=0x7f19d4138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 >> v1:172.21.15.53:6789/0 conn(0x7f19d413f710 legacy=0x7f19d41d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 1 monclient: found mon.a 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d41377c0 con 0x7f19d40a98f0 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:37.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 5 monclient: authenticate success, global_id 4305 2024-03-25T19:42:37.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 10 monclient: _renew_subs 2024-03-25T19:42:37.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:37.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d4137a70 con 0x7f19d40a98f0 2024-03-25T19:42:37.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:37.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 10 monclient: _renew_subs 2024-03-25T19:42:37.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:37.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d41341f0 con 0x7f19d40a98f0 2024-03-25T19:42:37.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.074+0000 7f19d8c57640 1 librados: init done 2024-03-25T19:42:37.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.075+0000 7f19d8c57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:37.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.075+0000 7f19d8c57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:37.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.075+0000 7f19d8c57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:37.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.075+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d40abbe0 con 0x7f19d40a98f0 2024-03-25T19:42:37.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.379+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19c4001c50 con 0x7f19d40a98f0 2024-03-25T19:42:37.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.379+0000 7f19c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:37.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.379+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f19c4003a60 con 0x7f19d40a98f0 2024-03-25T19:42:37.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.379+0000 7f19c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.379+0000 7f19c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:37.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.379+0000 7f19c2ffd640 10 monclient: dump: 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:37.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.379+0000 7f19d0d99640 4 set_mon_vals no callback set 2024-03-25T19:42:37.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.380+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f19c4003c80 con 0x7f19d40a98f0 2024-03-25T19:42:37.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.381+0000 7f19d1d9b640 10 monclient: get_auth_request con 0x7f19b406de60 auth_method 0 2024-03-25T19:42:37.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.381+0000 7f19c2ffd640 10 client.4305.objecter ms_handle_connect 0x7f19b406de60 2024-03-25T19:42:37.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.388+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f19c4054810 con 0x7f19d40a98f0 2024-03-25T19:42:37.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.388+0000 7f19c2ffd640 10 client.4305.objecter ms_dispatch 0x7f19d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:37.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.388+0000 7f19c2ffd640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:37.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.388+0000 7f19c2ffd640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:37.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.388+0000 7f19c2ffd640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T19:42:37.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.388+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19c4050320 con 0x7f19d40a98f0 2024-03-25T19:42:37.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.388+0000 7f19c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:37.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.388+0000 7f19c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:37.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.644+0000 7f19d8c57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.644+0000 7f19d8c57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.644+0000 7f19d8c57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:37.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.644+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19d40627b0 con 0x7f19d40a98f0 2024-03-25T19:42:37.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.676+0000 7f19c2ffd640 1 -- 172.21.15.18:0/2861627069 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f19c4052250 con 0x7f19d40a98f0 2024-03-25T19:42:37.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.676+0000 7f19c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.676+0000 7f19c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:37.677 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:42:37.677 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:37.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.678+0000 7f19d8c57640 10 librados: watch_flush enter 2024-03-25T19:42:37.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.678+0000 7f19d8c57640 10 librados: watch_flush exit 2024-03-25T19:42:37.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.678+0000 7f19d8c57640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T19:42:37.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.678+0000 7f19d8c57640 10 client.4305.objecter successfully canceled tick 2024-03-25T19:42:37.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.678+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 >> v1:172.21.15.18:6800/1520000573 conn(0x7f19b406de60 legacy=0x7f19b4070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.678+0000 7f19d8c57640 10 monclient: shutdown 2024-03-25T19:42:37.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.678+0000 7f19d8c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:37.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.678+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 >> v1:172.21.15.18:6789/0 conn(0x7f19d40a98f0 legacy=0x7f19d4134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.679+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 shutdown_connections 2024-03-25T19:42:37.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.679+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 >> 172.21.15.18:0/2861627069 conn(0x7f19d40b7ea0 msgr2=0x7f19d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:37.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.679+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 shutdown_connections 2024-03-25T19:42:37.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.679+0000 7f19d8c57640 1 -- 172.21.15.18:0/2861627069 wait complete. 2024-03-25T19:42:37.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:37.679+0000 7f19d8c57640 1 librados: shutdown 2024-03-25T19:42:37.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:44.498 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.616+0000 7f02e773d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.616+0000 7f02e773d640 10 monclient: build_initial_monmap 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.616+0000 7f02e773d640 10 monclient: monmap: 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:42:44.617324+0000 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:42:44.617324+0000 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:42:44.616 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:42:44.617 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:44.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.617+0000 7f02e773d640 1 Processor -- start 2024-03-25T19:42:44.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.617+0000 7f02e773d640 1 -- start start 2024-03-25T19:42:44.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.617+0000 7f02e773d640 10 monclient: init 2024-03-25T19:42:44.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:44.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:44.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 10 monclient(hunting): picked mon.noname-b con 0x7f02e00b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:44.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 10 monclient(hunting): picked mon.noname-a con 0x7f02e00b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:44.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 10 monclient(hunting): picked mon.noname-c con 0x7f02e014be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:44.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00af940 con 0x7f02e00b1730 2024-03-25T19:42:44.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00b0a70 con 0x7f02e014be90 2024-03-25T19:42:44.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e00b0d50 con 0x7f02e00b4d60 2024-03-25T19:42:44.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:44.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.618+0000 7f02e773d640 10 monclient(hunting): authenticate will time out at 4991.745069s 2024-03-25T19:42:44.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.621+0000 7f02e5cda640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f02e014be90 0x7f02e014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:33242/0 (socket says 172.21.15.18:33242) 2024-03-25T19:42:44.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.622+0000 7f02e5cda640 1 -- 172.21.15.18:0/2461885827 learned_addr learned my addr 172.21.15.18:0/2461885827 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:44.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.635+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833661773 0 0) 0x7f02e00b0a70 con 0x7f02e014be90 2024-03-25T19:42:44.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.635+0000 7f02d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.635+0000 7f02d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.635+0000 7f02d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.635+0000 7f02d7fff640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:42:44.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.635+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02cc003520 con 0x7f02e014be90 2024-03-25T19:42:44.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.643+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532729268 0 0) 0x7f02e00af940 con 0x7f02e00b1730 2024-03-25T19:42:44.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.643+0000 7f02d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.643+0000 7f02d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.643+0000 7f02d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.643+0000 7f02d7fff640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:42:44.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.643+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e00b0a70 con 0x7f02e00b1730 2024-03-25T19:42:44.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.654+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69101539 0 0) 0x7f02e00b0d50 con 0x7f02e00b4d60 2024-03-25T19:42:44.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.654+0000 7f02d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.654+0000 7f02d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.654+0000 7f02d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.654+0000 7f02d7fff640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:42:44.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.654+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e00af940 con 0x7f02e00b4d60 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800302153 0 0) 0x7f02cc003520 con 0x7f02e014be90 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 >> v1:172.21.15.18:6790/0 conn(0x7f02e00b4d60 legacy=0x7f02e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 >> v1:172.21.15.18:6789/0 conn(0x7f02e00b1730 legacy=0x7f02e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 1 monclient: found mon.noname-c 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e00b1030 con 0x7f02e014be90 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:44.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f02d8001a10 con 0x7f02e014be90 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02d7fff640 10 monclient: dump: 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:44.656 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:44.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02e773d640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:42:44.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.655+0000 7f02e773d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:44.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02d8001b70 con 0x7f02e014be90 2024-03-25T19:42:44.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02d7fff640 1 -- 172.21.15.18:0/2461885827 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f02d8001f80 con 0x7f02e014be90 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02d7fff640 10 monclient: dump: 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02e773d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02e773d640 4 set_mon_vals no callback set 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02e773d640 10 monclient: shutdown 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02e773d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.665+0000 7f02e773d640 1 -- 172.21.15.18:0/2461885827 >> v1:172.21.15.53:6789/0 conn(0x7f02e014be90 legacy=0x7f02e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.666+0000 7f02e773d640 1 -- 172.21.15.18:0/2461885827 shutdown_connections 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.666+0000 7f02e773d640 1 -- 172.21.15.18:0/2461885827 >> 172.21.15.18:0/2461885827 conn(0x7f02e00617a0 msgr2=0x7f02e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.666+0000 7f02e773d640 1 -- 172.21.15.18:0/2461885827 shutdown_connections 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.666+0000 7f02e773d640 1 -- 172.21.15.18:0/2461885827 wait complete. 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 10 monclient: build_initial_monmap 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 10 monclient: monmap: 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:42:44.669 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 1 librados: starting msgr at 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 1 librados: starting objecter 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 1 Processor -- start 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 1 -- start start 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 1 librados: setting wanted keys 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 1 librados: calling monclient init 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.667+0000 7f02e773d640 10 monclient: init 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 10 monclient(hunting): picked mon.noname-c con 0x7f02e00b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 10 monclient(hunting): picked mon.noname-a con 0x7f02e00b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 10 monclient(hunting): picked mon.noname-b con 0x7f02e014be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e0140380 con 0x7f02e00b4d60 2024-03-25T19:42:44.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e0140660 con 0x7f02e014be90 2024-03-25T19:42:44.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e0141e30 con 0x7f02e00b1730 2024-03-25T19:42:44.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:44.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.668+0000 7f02e773d640 10 monclient(hunting): authenticate will time out at 4991.795634s 2024-03-25T19:42:44.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.679+0000 7f02e4cd8640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f02e00b4d60 0x7f02e0142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:45078/0 (socket says 172.21.15.18:45078) 2024-03-25T19:42:44.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.679+0000 7f02e4cd8640 1 -- 172.21.15.18:0/394366057 learned_addr learned my addr 172.21.15.18:0/394366057 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:44.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.680+0000 7f02d5ffb640 10 client.?.objecter ms_handle_connect 0x7f02e00b4d60 2024-03-25T19:42:44.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.680+0000 7f02d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.689+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480122971 0 0) 0x7f02e0140380 con 0x7f02e00b4d60 2024-03-25T19:42:44.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.689+0000 7f02d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.689+0000 7f02d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.689+0000 7f02d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.689+0000 7f02d5ffb640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:42:44.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.689+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c4004eb0 con 0x7f02e00b4d60 2024-03-25T19:42:44.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.690+0000 7f02d5ffb640 10 client.?.objecter ms_handle_connect 0x7f02e00b1730 2024-03-25T19:42:44.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.690+0000 7f02d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.693+0000 7f02d5ffb640 10 client.?.objecter ms_handle_connect 0x7f02e014be90 2024-03-25T19:42:44.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.693+0000 7f02d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.703+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777261189 0 0) 0x7f02e0141e30 con 0x7f02e00b1730 2024-03-25T19:42:44.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.703+0000 7f02d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.703+0000 7f02d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.703+0000 7f02d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.703+0000 7f02d5ffb640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:42:44.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.703+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e0140380 con 0x7f02e00b1730 2024-03-25T19:42:44.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.704+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833416900 0 0) 0x7f02e0140660 con 0x7f02e014be90 2024-03-25T19:42:44.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.704+0000 7f02d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.704+0000 7f02d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.704+0000 7f02d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.704+0000 7f02d5ffb640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:42:44.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.704+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e0141e30 con 0x7f02e014be90 2024-03-25T19:42:44.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.710+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 863783747 0 0) 0x7f02c4004eb0 con 0x7f02e00b4d60 2024-03-25T19:42:44.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.710+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02e0140660 con 0x7f02e00b4d60 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.710+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f02d0001a30 con 0x7f02e00b4d60 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.710+0000 7f02d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.710+0000 7f02d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.710+0000 7f02d5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:44.710 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:44.711 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:44.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.710+0000 7f02d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3867667586 0 0) 0x7f02e0140660 con 0x7f02e00b4d60 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 >> v1:172.21.15.18:6790/0 conn(0x7f02e00b1730 legacy=0x7f02e0138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 >> v1:172.21.15.53:6789/0 conn(0x7f02e014be90 legacy=0x7f02e01e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 1 monclient: found mon.a 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e0142110 con 0x7f02e00b4d60 2024-03-25T19:42:44.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 5 monclient: authenticate success, global_id 4311 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 10 monclient: _renew_subs 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e0140e60 con 0x7f02e00b4d60 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 10 monclient: _renew_subs 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e0142c40 con 0x7f02e00b4d60 2024-03-25T19:42:44.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.725+0000 7f02e773d640 1 librados: init done 2024-03-25T19:42:44.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.726+0000 7f02e773d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.726+0000 7f02e773d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.726+0000 7f02e773d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:44.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:44.726+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e00627b0 con 0x7f02e00b4d60 2024-03-25T19:42:45.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.041+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02d0001c70 con 0x7f02e00b4d60 2024-03-25T19:42:45.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.041+0000 7f02d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:45.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.041+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f02d0003a60 con 0x7f02e00b4d60 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.041+0000 7f02d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.041+0000 7f02d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.041+0000 7f02d5ffb640 10 monclient: dump: 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:45.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.041+0000 7f02d77fe640 4 set_mon_vals no callback set 2024-03-25T19:42:45.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.042+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f02d0003c40 con 0x7f02e00b4d60 2024-03-25T19:42:45.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.043+0000 7f02e54d9640 10 monclient: get_auth_request con 0x7f02c406ddf0 auth_method 0 2024-03-25T19:42:45.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.043+0000 7f02d5ffb640 10 client.4311.objecter ms_handle_connect 0x7f02c406ddf0 2024-03-25T19:42:45.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.052+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f02d00548d0 con 0x7f02e00b4d60 2024-03-25T19:42:45.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.052+0000 7f02d5ffb640 10 client.4311.objecter ms_dispatch 0x7f02e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:45.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.052+0000 7f02d5ffb640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:45.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.052+0000 7f02d5ffb640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:45.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.052+0000 7f02d5ffb640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-25T19:42:45.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.052+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02d00503e0 con 0x7f02e00b4d60 2024-03-25T19:42:45.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.052+0000 7f02d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:45.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.052+0000 7f02d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:45.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.310+0000 7f02e773d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.310+0000 7f02e773d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.310+0000 7f02e773d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:42:45.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.310+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02e0140810 con 0x7f02e00b4d60 2024-03-25T19:42:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.341+0000 7f02d5ffb640 1 -- 172.21.15.18:0/394366057 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f02e0140810 con 0x7f02e00b4d60 2024-03-25T19:42:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.341+0000 7f02d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.341+0000 7f02d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:45.341 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:42:45.341 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 10 librados: watch_flush enter 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 10 librados: watch_flush exit 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 10 client.4311.objecter successfully canceled tick 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 >> v1:172.21.15.18:6800/1520000573 conn(0x7f02c406ddf0 legacy=0x7f02c40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 10 monclient: shutdown 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 >> v1:172.21.15.18:6789/0 conn(0x7f02e00b4d60 legacy=0x7f02e0142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 shutdown_connections 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 >> 172.21.15.18:0/394366057 conn(0x7f02e00617a0 msgr2=0x7f02e00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 shutdown_connections 2024-03-25T19:42:45.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 1 -- 172.21.15.18:0/394366057 wait complete. 2024-03-25T19:42:45.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:45.344+0000 7f02e773d640 1 librados: shutdown 2024-03-25T19:42:45.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:46.985 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:14.515 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:47.043 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:14.574 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:47.086 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:14.617 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:47.095 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:14.592 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:47.097 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:14.628 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:47.101 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:14.632 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:47.103 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:14.599 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:47.124 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:14.621 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:47.130 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:14.661 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:52.162 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:52.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.282+0000 7ff120ca1640 10 monclient: get_monmap_and_config 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.282+0000 7ff120ca1640 10 monclient: build_initial_monmap 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.282+0000 7ff120ca1640 10 monclient: monmap: 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:42:52.283054+0000 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:42:52.283054+0000 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:52.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.283+0000 7ff120ca1640 1 Processor -- start 2024-03-25T19:42:52.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.283+0000 7ff120ca1640 1 -- start start 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.283+0000 7ff120ca1640 10 monclient: init 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.283+0000 7ff120ca1640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.283+0000 7ff120ca1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.283+0000 7ff120ca1640 10 monclient(hunting): picked mon.noname-a con 0x7ff11c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.284+0000 7ff120ca1640 10 monclient(hunting): picked mon.noname-c con 0x7ff11c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.284+0000 7ff120ca1640 10 monclient(hunting): picked mon.noname-b con 0x7ff11c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.284+0000 7ff120ca1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c142520 con 0x7ff11c0a98f0 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.284+0000 7ff120ca1640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c0b28a0 con 0x7ff11c13f710 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.284+0000 7ff120ca1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c0b2a30 con 0x7ff11c1430c0 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.284+0000 7ff120ca1640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:52.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.284+0000 7ff120ca1640 10 monclient(hunting): authenticate will time out at 4999.410927s 2024-03-25T19:42:52.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.286+0000 7ff113fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff11c13f710 0x7ff11c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:34464/0 (socket says 172.21.15.18:34464) 2024-03-25T19:42:52.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.286+0000 7ff113fff640 1 -- 172.21.15.18:0/1053908239 learned_addr learned my addr 172.21.15.18:0/1053908239 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:52.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.296+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361960477 0 0) 0x7ff11c0b28a0 con 0x7ff11c13f710 2024-03-25T19:42:52.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.296+0000 7ff119d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.296+0000 7ff119d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.296+0000 7ff119d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.296+0000 7ff119d9b640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:42:52.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.296+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff108003520 con 0x7ff11c13f710 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.312+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162324375 0 0) 0x7ff11c142520 con 0x7ff11c0a98f0 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.313+0000 7ff119d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.313+0000 7ff119d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.313+0000 7ff119d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.313+0000 7ff119d9b640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.313+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c0b28a0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699499746 0 0) 0x7ff108003520 con 0x7ff11c13f710 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 >> v1:172.21.15.18:6790/0 conn(0x7ff11c1430c0 legacy=0x7ff11c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 >> v1:172.21.15.18:6789/0 conn(0x7ff11c0a98f0 legacy=0x7ff11c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 1 monclient: found mon.noname-c 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c0b44c0 con 0x7ff11c13f710 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff104001690 con 0x7ff11c13f710 2024-03-25T19:42:52.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff119d9b640 10 monclient: dump: 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff120ca1640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.320+0000 7ff120ca1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:52.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1040017f0 con 0x7ff11c13f710 2024-03-25T19:42:52.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff119d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:52.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff119d9b640 1 -- 172.21.15.18:0/1053908239 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff104001c00 con 0x7ff11c13f710 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff119d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff119d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff119d9b640 10 monclient: dump: 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff120ca1640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff120ca1640 4 set_mon_vals no callback set 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff120ca1640 10 monclient: shutdown 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff120ca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.340+0000 7ff120ca1640 1 -- 172.21.15.18:0/1053908239 >> v1:172.21.15.53:6789/0 conn(0x7ff11c13f710 legacy=0x7ff11c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.341+0000 7ff120ca1640 1 -- 172.21.15.18:0/1053908239 shutdown_connections 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.341+0000 7ff120ca1640 1 -- 172.21.15.18:0/1053908239 >> 172.21.15.18:0/1053908239 conn(0x7ff11c0b7ea0 msgr2=0x7ff11c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.341+0000 7ff120ca1640 1 -- 172.21.15.18:0/1053908239 shutdown_connections 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.341+0000 7ff120ca1640 1 -- 172.21.15.18:0/1053908239 wait complete. 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 10 monclient: build_initial_monmap 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 10 monclient: monmap: 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 1 librados: starting msgr at 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 1 librados: starting objecter 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 1 Processor -- start 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 1 -- start start 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 1 librados: setting wanted keys 2024-03-25T19:42:52.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 1 librados: calling monclient init 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 10 monclient: init 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.342+0000 7ff120ca1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.343+0000 7ff120ca1640 10 monclient(hunting): picked mon.noname-b con 0x7ff11c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.343+0000 7ff120ca1640 10 monclient(hunting): picked mon.noname-c con 0x7ff11c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.343+0000 7ff120ca1640 10 monclient(hunting): picked mon.noname-a con 0x7ff11c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.343+0000 7ff120ca1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c0abdc0 con 0x7ff11c1430c0 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.343+0000 7ff120ca1640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c1d7730 con 0x7ff11c0a98f0 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.343+0000 7ff120ca1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c138f10 con 0x7ff11c13f710 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.343+0000 7ff120ca1640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:52.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.343+0000 7ff120ca1640 10 monclient(hunting): authenticate will time out at 4999.469979s 2024-03-25T19:42:52.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.352+0000 7ff11a59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff11c0a98f0 0x7ff11c138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:34516/0 (socket says 172.21.15.18:34516) 2024-03-25T19:42:52.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.352+0000 7ff11a59c640 1 -- 172.21.15.18:0/1488522251 learned_addr learned my addr 172.21.15.18:0/1488522251 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:52.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.353+0000 7ff112ffd640 10 client.?.objecter ms_handle_connect 0x7ff11c0a98f0 2024-03-25T19:42:52.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.353+0000 7ff112ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.359+0000 7ff112ffd640 10 client.?.objecter ms_handle_connect 0x7ff11c13f710 2024-03-25T19:42:52.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.359+0000 7ff112ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.370+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605693210 0 0) 0x7ff11c1d7730 con 0x7ff11c0a98f0 2024-03-25T19:42:52.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.370+0000 7ff112ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.370+0000 7ff112ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.370+0000 7ff112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.370+0000 7ff112ffd640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:42:52.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.370+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc004eb0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.380+0000 7ff112ffd640 10 client.?.objecter ms_handle_connect 0x7ff11c1430c0 2024-03-25T19:42:52.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.380+0000 7ff112ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.383+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847167837 0 0) 0x7ff11c138f10 con 0x7ff11c13f710 2024-03-25T19:42:52.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.383+0000 7ff112ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.383+0000 7ff112ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.383+0000 7ff112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.383+0000 7ff112ffd640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:42:52.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.383+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c1d7730 con 0x7ff11c13f710 2024-03-25T19:42:52.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.394+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481473554 0 0) 0x7ff11c0abdc0 con 0x7ff11c1430c0 2024-03-25T19:42:52.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.395+0000 7ff112ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.395+0000 7ff112ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.395+0000 7ff112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.395+0000 7ff112ffd640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:42:52.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.395+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c138f10 con 0x7ff11c1430c0 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.397+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1489049292 0 0) 0x7ff0fc004eb0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.397+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff11c0abdc0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.397+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff10c001a10 con 0x7ff11c0a98f0 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.397+0000 7ff112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.397+0000 7ff112ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.397+0000 7ff112ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.397 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.398 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.398 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:52.398 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:52.398 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:52.398 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:52.398 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:52.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.397+0000 7ff112ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:52.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.411+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1239965829 0 0) 0x7ff11c1d7730 con 0x7ff11c13f710 2024-03-25T19:42:52.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.412+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0fc004eb0 con 0x7ff11c13f710 2024-03-25T19:42:52.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.412+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff104001b00 con 0x7ff11c13f710 2024-03-25T19:42:52.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.412+0000 7ff112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.412+0000 7ff112ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.412+0000 7ff112ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:52.412 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:52.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.413+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2480543 0 0) 0x7ff11c138f10 con 0x7ff11c1430c0 2024-03-25T19:42:52.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.413+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff11c1d7730 con 0x7ff11c1430c0 2024-03-25T19:42:52.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.413+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff114001a90 con 0x7ff11c1430c0 2024-03-25T19:42:52.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.413+0000 7ff112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.413+0000 7ff112ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:52.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.413+0000 7ff112ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:52.413 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:52.413 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:52.414 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.414 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.414 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.414 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:52.414 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:52.414 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:52.414 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:52.414 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4261881685 0 0) 0x7ff11c0abdc0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 >> v1:172.21.15.18:6790/0 conn(0x7ff11c13f710 legacy=0x7ff11c1d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 >> v1:172.21.15.18:6789/0 conn(0x7ff11c1430c0 legacy=0x7ff11c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 1 monclient: found mon.b 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c1390a0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff112ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 5 monclient: authenticate success, global_id 4291 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 10 monclient: _renew_subs 2024-03-25T19:42:52.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11c1d8320 con 0x7ff11c0a98f0 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 10 monclient: _renew_subs 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11c1d43c0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.415+0000 7ff120ca1640 1 librados: init done 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.416+0000 7ff120ca1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.416+0000 7ff120ca1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.416+0000 7ff120ca1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:52.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.416+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11c0627b0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.720+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff10c001c50 con 0x7ff11c0a98f0 2024-03-25T19:42:52.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.721+0000 7ff112ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.721+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff10c003a60 con 0x7ff11c0a98f0 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.721+0000 7ff112ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.721+0000 7ff112ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.721+0000 7ff112ffd640 10 monclient: dump: 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:42:52.721 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:42:52.722 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:52.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.721+0000 7ff11959a640 4 set_mon_vals no callback set 2024-03-25T19:42:52.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.721+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ff10c003c80 con 0x7ff11c0a98f0 2024-03-25T19:42:52.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.722+0000 7ff113fff640 10 monclient: get_auth_request con 0x7ff0fc06e0f0 auth_method 0 2024-03-25T19:42:52.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.722+0000 7ff112ffd640 10 client.4291.objecter ms_handle_connect 0x7ff0fc06e0f0 2024-03-25T19:42:52.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.727+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7ff10c0548a0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.727+0000 7ff112ffd640 10 client.4291.objecter ms_dispatch 0x7ff11c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:52.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.727+0000 7ff112ffd640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:52.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.727+0000 7ff112ffd640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:52.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.727+0000 7ff112ffd640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T19:42:52.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.728+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff10c0503b0 con 0x7ff11c0a98f0 2024-03-25T19:42:52.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.728+0000 7ff112ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.728+0000 7ff112ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:52.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.968+0000 7ff120ca1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:52.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.968+0000 7ff120ca1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:52.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.968+0000 7ff120ca1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:42:52.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:52.968+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff11c069f40 con 0x7ff11c0a98f0 2024-03-25T19:42:53.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.003+0000 7ff112ffd640 1 -- 172.21.15.18:0/1488522251 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7ff10c0522e0 con 0x7ff11c0a98f0 2024-03-25T19:42:53.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.003+0000 7ff112ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:53.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.003+0000 7ff112ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:53.003 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:42:53.003 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.005+0000 7ff120ca1640 10 librados: watch_flush enter 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.005+0000 7ff120ca1640 10 librados: watch_flush exit 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.005+0000 7ff120ca1640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.005+0000 7ff120ca1640 10 client.4291.objecter successfully canceled tick 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.005+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 >> v1:172.21.15.18:6800/1520000573 conn(0x7ff0fc06e0f0 legacy=0x7ff0fc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.005+0000 7ff120ca1640 10 monclient: shutdown 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.005+0000 7ff120ca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.005+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 >> v1:172.21.15.53:6789/0 conn(0x7ff11c0a98f0 legacy=0x7ff11c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.006+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 shutdown_connections 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.006+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 >> 172.21.15.18:0/1488522251 conn(0x7ff11c0b7ea0 msgr2=0x7ff11c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.006+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 shutdown_connections 2024-03-25T19:42:53.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.006+0000 7ff120ca1640 1 -- 172.21.15.18:0/1488522251 wait complete. 2024-03-25T19:42:53.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:53.006+0000 7ff120ca1640 1 librados: shutdown 2024-03-25T19:42:53.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:59.824 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.929+0000 7ffb82f79640 10 monclient: get_monmap_and_config 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.929+0000 7ffb82f79640 10 monclient: build_initial_monmap 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.929+0000 7ffb82f79640 10 monclient: monmap: 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:42:59.930013+0000 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:42:59.930013+0000 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:42:59.929 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:42:59.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.929+0000 7ffb82f79640 1 Processor -- start 2024-03-25T19:42:59.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.929+0000 7ffb82f79640 1 -- start start 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.929+0000 7ffb82f79640 10 monclient: init 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.930+0000 7ffb82f79640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.930+0000 7ffb82f79640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.930+0000 7ffb82f79640 10 monclient(hunting): picked mon.noname-a con 0x7ffb7c0b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.930+0000 7ffb82f79640 10 monclient(hunting): picked mon.noname-c con 0x7ffb7c0b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.931+0000 7ffb82f79640 10 monclient(hunting): picked mon.noname-b con 0x7ffb7c14be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.931+0000 7ffb82f79640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c0af940 con 0x7ffb7c0b4d60 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.931+0000 7ffb82f79640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c0b0a70 con 0x7ffb7c0b1730 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.931+0000 7ffb82f79640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c0b0d50 con 0x7ffb7c14be90 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.931+0000 7ffb82f79640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.931+0000 7ffb82f79640 10 monclient(hunting): authenticate will time out at 5007.057815s 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.931+0000 7ffb81516640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ffb7c14be90 0x7ffb7c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:52812/0 (socket says 172.21.15.18:52812) 2024-03-25T19:42:59.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.931+0000 7ffb81516640 1 -- 172.21.15.18:0/2572137061 learned_addr learned my addr 172.21.15.18:0/2572137061 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:42:59.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.998+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887497896 0 0) 0x7ffb7c0af940 con 0x7ffb7c0b4d60 2024-03-25T19:42:59.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.998+0000 7ffb7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.998+0000 7ffb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.998+0000 7ffb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.998+0000 7ffb7b7fe640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:42:59.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:42:59.998+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64003520 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 505686464 0 0) 0x7ffb64003520 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 >> v1:172.21.15.18:6790/0 conn(0x7ffb7c14be90 legacy=0x7ffb7c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 >> v1:172.21.15.53:6789/0 conn(0x7ffb7c0b1730 legacy=0x7ffb7c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7c0b1030 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffb6c001690 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb82f79640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb82f79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb7b7fe640 10 monclient: dump: 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:00.010 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.009+0000 7ffb82f79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:00.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.020+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb6c0017f0 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.020+0000 7ffb7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:00.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.020+0000 7ffb7b7fe640 1 -- 172.21.15.18:0/2572137061 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffb6c001c00 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.020+0000 7ffb7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.020+0000 7ffb7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.020+0000 7ffb7b7fe640 10 monclient: dump: 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.020+0000 7ffb82f79640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:00.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.020+0000 7ffb82f79640 4 set_mon_vals no callback set 2024-03-25T19:43:00.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 10 monclient: shutdown 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 1 -- 172.21.15.18:0/2572137061 >> v1:172.21.15.18:6789/0 conn(0x7ffb7c0b4d60 legacy=0x7ffb7c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 1 -- 172.21.15.18:0/2572137061 shutdown_connections 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 1 -- 172.21.15.18:0/2572137061 >> 172.21.15.18:0/2572137061 conn(0x7ffb7c0617a0 msgr2=0x7ffb7c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 1 -- 172.21.15.18:0/2572137061 shutdown_connections 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 1 -- 172.21.15.18:0/2572137061 wait complete. 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 10 monclient: build_initial_monmap 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.021+0000 7ffb82f79640 10 monclient: monmap: 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.022+0000 7ffb82f79640 1 librados: starting msgr at 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.022+0000 7ffb82f79640 1 librados: starting objecter 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.022+0000 7ffb82f79640 1 Processor -- start 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.022+0000 7ffb82f79640 1 -- start start 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.022+0000 7ffb82f79640 1 librados: setting wanted keys 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.022+0000 7ffb82f79640 1 librados: calling monclient init 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.022+0000 7ffb82f79640 10 monclient: init 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 10 monclient(hunting): picked mon.noname-c con 0x7ffb7c0b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 10 monclient(hunting): picked mon.noname-b con 0x7ffb7c0b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 10 monclient(hunting): picked mon.noname-a con 0x7ffb7c14be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c13a940 con 0x7ffb7c14be90 2024-03-25T19:43:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c13ab90 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7c1d7840 con 0x7ffb7c0b1730 2024-03-25T19:43:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.023+0000 7ffb82f79640 10 monclient(hunting): authenticate will time out at 5007.150042s 2024-03-25T19:43:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.030+0000 7ffb7bfff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ffb7c0b4d60 0x7ffb7c1d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:53088/0 (socket says 172.21.15.18:53088) 2024-03-25T19:43:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.030+0000 7ffb7bfff640 1 -- 172.21.15.18:0/2369441630 learned_addr learned my addr 172.21.15.18:0/2369441630 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:00.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.030+0000 7ffb797fa640 10 client.?.objecter ms_handle_connect 0x7ffb7c0b4d60 2024-03-25T19:43:00.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.030+0000 7ffb797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:00.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.040+0000 7ffb797fa640 10 client.?.objecter ms_handle_connect 0x7ffb7c14be90 2024-03-25T19:43:00.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.040+0000 7ffb797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:00.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.051+0000 7ffb797fa640 10 client.?.objecter ms_handle_connect 0x7ffb7c0b1730 2024-03-25T19:43:00.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.051+0000 7ffb797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:00.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.062+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349032241 0 0) 0x7ffb7c13a940 con 0x7ffb7c14be90 2024-03-25T19:43:00.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.062+0000 7ffb797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.062+0000 7ffb797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.062+0000 7ffb797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.062+0000 7ffb797fa640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:43:00.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.063+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb68004eb0 con 0x7ffb7c14be90 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.136+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3703757270 0 0) 0x7ffb68004eb0 con 0x7ffb7c14be90 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.136+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb7c13a940 con 0x7ffb7c14be90 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.136+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffb74001b00 con 0x7ffb7c14be90 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.136+0000 7ffb797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.136+0000 7ffb797fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.136+0000 7ffb797fa640 10 monclient(hunting): dump: 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:00.136 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:00.137 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:00.137 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:00.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.136+0000 7ffb797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:00.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.140+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498497971 0 0) 0x7ffb7c1d7840 con 0x7ffb7c0b1730 2024-03-25T19:43:00.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.140+0000 7ffb797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.140+0000 7ffb797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.140+0000 7ffb797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.140+0000 7ffb797fa640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:43:00.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.140+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb68004eb0 con 0x7ffb7c0b1730 2024-03-25T19:43:00.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.148+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720847056 0 0) 0x7ffb7c13ab90 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.148+0000 7ffb797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.148+0000 7ffb797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.148+0000 7ffb797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.148+0000 7ffb797fa640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:43:00.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.148+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb7c1d7840 con 0x7ffb7c0b4d60 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1400490720 0 0) 0x7ffb7c13a940 con 0x7ffb7c14be90 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 >> v1:172.21.15.18:6790/0 conn(0x7ffb7c0b1730 legacy=0x7ffb7c13bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 >> v1:172.21.15.53:6789/0 conn(0x7ffb7c0b4d60 legacy=0x7ffb7c1d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 1 monclient: found mon.a 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7c13c240 con 0x7ffb7c14be90 2024-03-25T19:43:00.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 5 monclient: authenticate success, global_id 4323 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 10 monclient: _renew_subs 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb7c139ce0 con 0x7ffb7c14be90 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 10 monclient: _renew_subs 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb7c1d8650 con 0x7ffb7c14be90 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.152+0000 7ffb82f79640 1 librados: init done 2024-03-25T19:43:00.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.153+0000 7ffb82f79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.153+0000 7ffb82f79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.153+0000 7ffb82f79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:00.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.153+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb7c1401f0 con 0x7ffb7c14be90 2024-03-25T19:43:00.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.483+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb74001fb0 con 0x7ffb7c14be90 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.483+0000 7ffb797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.483+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffb74004d00 con 0x7ffb7c14be90 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.483+0000 7ffb797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.483+0000 7ffb797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.483+0000 7ffb797fa640 10 monclient: dump: 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:00.483 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:00.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.483+0000 7ffb7affd640 4 set_mon_vals no callback set 2024-03-25T19:43:00.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.483+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ffb74004f20 con 0x7ffb7c14be90 2024-03-25T19:43:00.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.484+0000 7ffb80d15640 10 monclient: get_auth_request con 0x7ffb6806de60 auth_method 0 2024-03-25T19:43:00.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.484+0000 7ffb797fa640 10 client.4323.objecter ms_handle_connect 0x7ffb6806de60 2024-03-25T19:43:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.491+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7ffb74054910 con 0x7ffb7c14be90 2024-03-25T19:43:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.491+0000 7ffb797fa640 10 client.4323.objecter ms_dispatch 0x7ffb7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.491+0000 7ffb797fa640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.491+0000 7ffb797fa640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.491+0000 7ffb797fa640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T19:43:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.491+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb74050420 con 0x7ffb7c14be90 2024-03-25T19:43:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.491+0000 7ffb797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.491+0000 7ffb797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:00.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.743+0000 7ffb82f79640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.743+0000 7ffb82f79640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.743+0000 7ffb82f79640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:00.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.743+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb7c0627b0 con 0x7ffb7c14be90 2024-03-25T19:43:00.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.770+0000 7ffb797fa640 1 -- 172.21.15.18:0/2369441630 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7ffb74052350 con 0x7ffb7c14be90 2024-03-25T19:43:00.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.770+0000 7ffb797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.770+0000 7ffb797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:00.770 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:43:00.770 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:00.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 10 librados: watch_flush enter 2024-03-25T19:43:00.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 10 librados: watch_flush exit 2024-03-25T19:43:00.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T19:43:00.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 10 client.4323.objecter successfully canceled tick 2024-03-25T19:43:00.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 >> v1:172.21.15.18:6800/1520000573 conn(0x7ffb6806de60 legacy=0x7ffb68070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 10 monclient: shutdown 2024-03-25T19:43:00.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:00.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 >> v1:172.21.15.18:6789/0 conn(0x7ffb7c14be90 legacy=0x7ffb7c1e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 shutdown_connections 2024-03-25T19:43:00.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 >> 172.21.15.18:0/2369441630 conn(0x7ffb7c0617a0 msgr2=0x7ffb7c137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:00.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 shutdown_connections 2024-03-25T19:43:00.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.772+0000 7ffb82f79640 1 -- 172.21.15.18:0/2369441630 wait complete. 2024-03-25T19:43:00.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:00.773+0000 7ffb82f79640 1 librados: shutdown 2024-03-25T19:43:00.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:07.028 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:34.559 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.076 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:34.607 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.116 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:34.647 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.124 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:34.620 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.143 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:34.640 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.149 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:34.680 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.158 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:34.689 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.181 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:34.678 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.188 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:34.719 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:07.590 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.702+0000 7f2a25fc7640 10 monclient: get_monmap_and_config 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.702+0000 7f2a25fc7640 10 monclient: build_initial_monmap 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.702+0000 7f2a25fc7640 10 monclient: monmap: 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:43:07.703358+0000 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:43:07.703358+0000 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:07.702 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:43:07.703 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:43:07.703 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 1 Processor -- start 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 1 -- start start 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 10 monclient: init 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 10 monclient(hunting): picked mon.noname-c con 0x7f2a200b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 10 monclient(hunting): picked mon.noname-a con 0x7f2a200b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 10 monclient(hunting): picked mon.noname-b con 0x7f2a2014be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a200af940 con 0x7f2a200b1730 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a200b0a70 con 0x7f2a200b4d60 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a200b0d50 con 0x7f2a2014be90 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.703+0000 7f2a25fc7640 10 monclient(hunting): authenticate will time out at 5014.830730s 2024-03-25T19:43:07.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.711+0000 7f2a1ffff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f2a2014be90 0x7f2a2014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:60012/0 (socket says 172.21.15.18:60012) 2024-03-25T19:43:07.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.711+0000 7f2a1ffff640 1 -- 172.21.15.18:0/584682249 learned_addr learned my addr 172.21.15.18:0/584682249 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.723+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840894390 0 0) 0x7f2a200b0d50 con 0x7f2a2014be90 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.723+0000 7f2a1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.723+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.723+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.723+0000 7f2a1e7fc640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:43:07.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.723+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a08003520 con 0x7f2a2014be90 2024-03-25T19:43:07.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.727+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565876260 0 0) 0x7f2a200af940 con 0x7f2a200b1730 2024-03-25T19:43:07.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.727+0000 7f2a1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.727+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.727+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.727+0000 7f2a1e7fc640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:43:07.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.727+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a200b0d50 con 0x7f2a200b1730 2024-03-25T19:43:07.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.734+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048936939 0 0) 0x7f2a200b0a70 con 0x7f2a200b4d60 2024-03-25T19:43:07.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.734+0000 7f2a1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.734+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.734+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.734+0000 7f2a1e7fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:43:07.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.734+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a200af940 con 0x7f2a200b4d60 2024-03-25T19:43:07.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.743+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516301984 0 0) 0x7f2a200af940 con 0x7f2a200b4d60 2024-03-25T19:43:07.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 >> v1:172.21.15.18:6790/0 conn(0x7f2a2014be90 legacy=0x7f2a2014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 >> v1:172.21.15.18:6789/0 conn(0x7f2a200b1730 legacy=0x7f2a200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a200b1030 con 0x7f2a200b4d60 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2a10001570 con 0x7f2a200b4d60 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a25fc7640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a25fc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a1e7fc640 10 monclient: dump: 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:07.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.744+0000 7f2a25fc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:07.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.777+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a100016d0 con 0x7f2a200b4d60 2024-03-25T19:43:07.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:07.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a1e7fc640 1 -- 172.21.15.18:0/584682249 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2a10001b20 con 0x7f2a200b4d60 2024-03-25T19:43:07.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:07.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 4 set_mon_vals no callback set 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a1e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a1e7fc640 10 monclient: dump: 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:07.778 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:07.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 10 monclient: shutdown 2024-03-25T19:43:07.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 1 -- 172.21.15.18:0/584682249 >> v1:172.21.15.53:6789/0 conn(0x7f2a200b4d60 legacy=0x7f2a2013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 1 -- 172.21.15.18:0/584682249 shutdown_connections 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 1 -- 172.21.15.18:0/584682249 >> 172.21.15.18:0/584682249 conn(0x7f2a200617a0 msgr2=0x7f2a20061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 1 -- 172.21.15.18:0/584682249 shutdown_connections 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.778+0000 7f2a25fc7640 1 -- 172.21.15.18:0/584682249 wait complete. 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.779+0000 7f2a25fc7640 10 monclient: build_initial_monmap 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.779+0000 7f2a25fc7640 10 monclient: monmap: 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:07.781 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.779+0000 7f2a25fc7640 1 librados: starting msgr at 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.779+0000 7f2a25fc7640 1 librados: starting objecter 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.780+0000 7f2a25fc7640 1 Processor -- start 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.780+0000 7f2a25fc7640 1 -- start start 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.780+0000 7f2a25fc7640 1 librados: setting wanted keys 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.780+0000 7f2a25fc7640 1 librados: calling monclient init 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.780+0000 7f2a25fc7640 10 monclient: init 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.780+0000 7f2a25fc7640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.780+0000 7f2a25fc7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.781+0000 7f2a25fc7640 10 monclient(hunting): picked mon.noname-c con 0x7f2a200b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.781+0000 7f2a25fc7640 10 monclient(hunting): picked mon.noname-b con 0x7f2a200b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.781+0000 7f2a25fc7640 10 monclient(hunting): picked mon.noname-a con 0x7f2a2014be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.781+0000 7f2a25fc7640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a20142c30 con 0x7f2a2014be90 2024-03-25T19:43:07.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.781+0000 7f2a25fc7640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a20139050 con 0x7f2a200b4d60 2024-03-25T19:43:07.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.781+0000 7f2a25fc7640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a20141540 con 0x7f2a200b1730 2024-03-25T19:43:07.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.781+0000 7f2a25fc7640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:07.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.781+0000 7f2a25fc7640 10 monclient(hunting): authenticate will time out at 5014.908076s 2024-03-25T19:43:07.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.787+0000 7f2a1effd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2a200b4d60 0x7f2a201d7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:39836/0 (socket says 172.21.15.18:39836) 2024-03-25T19:43:07.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.787+0000 7f2a1effd640 1 -- 172.21.15.18:0/1200200993 learned_addr learned my addr 172.21.15.18:0/1200200993 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.788+0000 7f2a07fff640 10 client.?.objecter ms_handle_connect 0x7f2a2014be90 2024-03-25T19:43:07.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.788+0000 7f2a07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.792+0000 7f2a07fff640 10 client.?.objecter ms_handle_connect 0x7f2a200b4d60 2024-03-25T19:43:07.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.792+0000 7f2a07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.792+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588125192 0 0) 0x7f2a20142c30 con 0x7f2a2014be90 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.792+0000 7f2a07fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.792+0000 7f2a07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.792+0000 7f2a07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.792+0000 7f2a07fff640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:43:07.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.792+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c004eb0 con 0x7f2a2014be90 2024-03-25T19:43:07.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.796+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980825612 0 0) 0x7f2a20139050 con 0x7f2a200b4d60 2024-03-25T19:43:07.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.796+0000 7f2a07fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.796+0000 7f2a07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.796+0000 7f2a07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.796+0000 7f2a07fff640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:43:07.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.796+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a20142c30 con 0x7f2a200b4d60 2024-03-25T19:43:07.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.797+0000 7f2a07fff640 10 client.?.objecter ms_handle_connect 0x7f2a200b1730 2024-03-25T19:43:07.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.797+0000 7f2a07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677174115 0 0) 0x7f2a20141540 con 0x7f2a200b1730 2024-03-25T19:43:07.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a20139050 con 0x7f2a200b1730 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371837371 0 0) 0x7f2a20142c30 con 0x7f2a200b4d60 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a20141540 con 0x7f2a200b4d60 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2a140019f0 con 0x7f2a200b4d60 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 10 monclient(hunting): dump: 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:07.805 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:07.806 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:07.806 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.806 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:07.806 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:07.806 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:07.806 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:07.806 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:07.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.804+0000 7f2a07fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:07.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2106061209 0 0) 0x7f2a20141540 con 0x7f2a200b4d60 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 >> v1:172.21.15.18:6790/0 conn(0x7f2a200b1730 legacy=0x7f2a20132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 >> v1:172.21.15.18:6789/0 conn(0x7f2a2014be90 legacy=0x7f2a201d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 1 monclient: found mon.b 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a20133630 con 0x7f2a200b4d60 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 10 monclient: _renew_subs 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2013ddd0 con 0x7f2a200b4d60 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:07.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 10 monclient: _renew_subs 2024-03-25T19:43:07.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:07.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a201422d0 con 0x7f2a200b4d60 2024-03-25T19:43:07.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.812+0000 7f2a25fc7640 1 librados: init done 2024-03-25T19:43:07.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.813+0000 7f2a25fc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.813+0000 7f2a25fc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.813+0000 7f2a25fc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:07.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:07.813+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2013e170 con 0x7f2a200b4d60 2024-03-25T19:43:08.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.097+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a140035f0 con 0x7f2a200b4d60 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.097+0000 7f2a07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.097+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2a14003940 con 0x7f2a200b4d60 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.097+0000 7f2a07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.097+0000 7f2a07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.097+0000 7f2a07fff640 10 monclient: dump: 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:08.097 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:08.098 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:08.098 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:08.098 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:08.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.097+0000 7f2a1dffb640 4 set_mon_vals no callback set 2024-03-25T19:43:08.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.098+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f2a14003b60 con 0x7f2a200b4d60 2024-03-25T19:43:08.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.098+0000 7f2a1f7fe640 10 monclient: get_auth_request con 0x7f2a0c06de20 auth_method 0 2024-03-25T19:43:08.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.099+0000 7f2a07fff640 10 client.4303.objecter ms_handle_connect 0x7f2a0c06de20 2024-03-25T19:43:08.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.104+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f2a14054800 con 0x7f2a200b4d60 2024-03-25T19:43:08.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.104+0000 7f2a07fff640 10 client.4303.objecter ms_dispatch 0x7f2a20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:08.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.104+0000 7f2a07fff640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:08.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.104+0000 7f2a07fff640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:08.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.104+0000 7f2a07fff640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-25T19:43:08.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.104+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a14050310 con 0x7f2a200b4d60 2024-03-25T19:43:08.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.104+0000 7f2a07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:08.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.104+0000 7f2a07fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:08.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.330+0000 7f2a25fc7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.330+0000 7f2a25fc7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.330+0000 7f2a25fc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:08.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.330+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a200627b0 con 0x7f2a200b4d60 2024-03-25T19:43:08.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.354+0000 7f2a07fff640 1 -- 172.21.15.18:0/1200200993 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f2a14052240 con 0x7f2a200b4d60 2024-03-25T19:43:08.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.354+0000 7f2a07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.354+0000 7f2a07fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:08.354 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:43:08.354 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:08.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 10 librados: watch_flush enter 2024-03-25T19:43:08.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 10 librados: watch_flush exit 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 10 client.4303.objecter successfully canceled tick 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 >> v1:172.21.15.18:6800/1520000573 conn(0x7f2a0c06de20 legacy=0x7f2a0c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 10 monclient: shutdown 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 >> v1:172.21.15.53:6789/0 conn(0x7f2a200b4d60 legacy=0x7f2a201d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 shutdown_connections 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 >> 172.21.15.18:0/1200200993 conn(0x7f2a200617a0 msgr2=0x7f2a200b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.356+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 shutdown_connections 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.357+0000 7f2a25fc7640 1 -- 172.21.15.18:0/1200200993 wait complete. 2024-03-25T19:43:08.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:08.357+0000 7f2a25fc7640 1 librados: shutdown 2024-03-25T19:43:08.368 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:15.174 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.290+0000 7f2edc196640 10 monclient: get_monmap_and_config 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.290+0000 7f2edc196640 10 monclient: build_initial_monmap 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.290+0000 7f2edc196640 10 monclient: monmap: 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:43:15.291285+0000 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:43:15.291285+0000 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:15.290 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:15.291 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:43:15.291 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:43:15.291 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:15.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.291+0000 7f2edc196640 1 Processor -- start 2024-03-25T19:43:15.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.291+0000 7f2edc196640 1 -- start start 2024-03-25T19:43:15.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.291+0000 7f2edc196640 10 monclient: init 2024-03-25T19:43:15.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:15.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:15.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 10 monclient(hunting): picked mon.noname-c con 0x7f2ed40a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:15.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 10 monclient(hunting): picked mon.noname-a con 0x7f2ed413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:15.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 10 monclient(hunting): picked mon.noname-b con 0x7f2ed41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:15.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed4142520 con 0x7f2ed413f710 2024-03-25T19:43:15.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40b28a0 con 0x7f2ed40a98f0 2024-03-25T19:43:15.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed40b2a30 con 0x7f2ed41430c0 2024-03-25T19:43:15.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:15.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.292+0000 7f2edc196640 10 monclient(hunting): authenticate will time out at 5022.419329s 2024-03-25T19:43:15.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.293+0000 7f2ed9f32640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2ed40a98f0 0x7f2ed40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:44374/0 (socket says 172.21.15.18:44374) 2024-03-25T19:43:15.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.293+0000 7f2ed9f32640 1 -- 172.21.15.18:0/3697732478 learned_addr learned my addr 172.21.15.18:0/3697732478 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:15.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.302+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438976489 0 0) 0x7f2ed40b28a0 con 0x7f2ed40a98f0 2024-03-25T19:43:15.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.302+0000 7f2ed8f30640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.302+0000 7f2ed8f30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.302+0000 7f2ed8f30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.302+0000 7f2ed8f30640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:43:15.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.302+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec0003520 con 0x7f2ed40a98f0 2024-03-25T19:43:15.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.313+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52336164 0 0) 0x7f2ed4142520 con 0x7f2ed413f710 2024-03-25T19:43:15.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.313+0000 7f2ed8f30640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.313+0000 7f2ed8f30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.313+0000 7f2ed8f30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.313+0000 7f2ed8f30640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:43:15.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.313+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed40b28a0 con 0x7f2ed413f710 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698944813 0 0) 0x7f2ec0003520 con 0x7f2ed40a98f0 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 >> v1:172.21.15.18:6790/0 conn(0x7f2ed41430c0 legacy=0x7f2ed414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 >> v1:172.21.15.18:6789/0 conn(0x7f2ed413f710 legacy=0x7f2ed41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 10 monclient: _finish_hunting 0 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 1 monclient: found mon.noname-c 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed40b44c0 con 0x7f2ed40a98f0 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 10 monclient: _finish_auth 0 2024-03-25T19:43:15.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 10 monclient: _check_auth_tickets 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2ed0001690 con 0x7f2ed40a98f0 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2ed8f30640 10 monclient: dump: 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:15.316 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:15.317 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:15.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2edc196640 5 monclient: authenticate success, global_id 4306 2024-03-25T19:43:15.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.315+0000 7f2edc196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.329+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ed00017f0 con 0x7f2ed40a98f0 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.329+0000 7f2ed8f30640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.329+0000 7f2ed8f30640 1 -- 172.21.15.18:0/3697732478 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2ed0001c00 con 0x7f2ed40a98f0 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.329+0000 7f2edc196640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.329+0000 7f2edc196640 4 set_mon_vals no callback set 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.329+0000 7f2ed8f30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.329+0000 7f2ed8f30640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.329+0000 7f2ed8f30640 10 monclient: dump: 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:15.329 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.330+0000 7f2edc196640 10 monclient: shutdown 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.330+0000 7f2edc196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.330+0000 7f2edc196640 1 -- 172.21.15.18:0/3697732478 >> v1:172.21.15.53:6789/0 conn(0x7f2ed40a98f0 legacy=0x7f2ed40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.330+0000 7f2edc196640 1 -- 172.21.15.18:0/3697732478 shutdown_connections 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.330+0000 7f2edc196640 1 -- 172.21.15.18:0/3697732478 >> 172.21.15.18:0/3697732478 conn(0x7f2ed40b7ea0 msgr2=0x7f2ed40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.330+0000 7f2edc196640 1 -- 172.21.15.18:0/3697732478 shutdown_connections 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.330+0000 7f2edc196640 1 -- 172.21.15.18:0/3697732478 wait complete. 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 10 monclient: build_initial_monmap 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 10 monclient: monmap: 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 1 librados: starting msgr at 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 1 librados: starting objecter 2024-03-25T19:43:15.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 1 Processor -- start 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 1 -- start start 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 1 librados: setting wanted keys 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 1 librados: calling monclient init 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.331+0000 7f2edc196640 10 monclient: init 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 10 monclient(hunting): picked mon.noname-b con 0x7f2ed40a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 10 monclient(hunting): picked mon.noname-a con 0x7f2ed413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 10 monclient(hunting): picked mon.noname-c con 0x7f2ed41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed41d7380 con 0x7f2ed413f710 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed4139840 con 0x7f2ed40a98f0 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed41d8200 con 0x7f2ed41430c0 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:15.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.332+0000 7f2edc196640 10 monclient(hunting): authenticate will time out at 5022.459457s 2024-03-25T19:43:15.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.358+0000 7f2ed9731640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f2ed413f710 0x7f2ed41da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:42350/0 (socket says 172.21.15.18:42350) 2024-03-25T19:43:15.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.358+0000 7f2ed9731640 1 -- 172.21.15.18:0/4072910365 learned_addr learned my addr 172.21.15.18:0/4072910365 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:15.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.361+0000 7f2eca7fc640 10 client.?.objecter ms_handle_connect 0x7f2ed41430c0 2024-03-25T19:43:15.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.361+0000 7f2eca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.365+0000 7f2eca7fc640 10 client.?.objecter ms_handle_connect 0x7f2ed413f710 2024-03-25T19:43:15.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.365+0000 7f2eca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.366+0000 7f2eca7fc640 10 client.?.objecter ms_handle_connect 0x7f2ed40a98f0 2024-03-25T19:43:15.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.366+0000 7f2eca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.378+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713548297 0 0) 0x7f2ed41d7380 con 0x7f2ed413f710 2024-03-25T19:43:15.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.378+0000 7f2eca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.378+0000 7f2eca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.378+0000 7f2eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.378+0000 7f2eca7fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:43:15.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.378+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb8004eb0 con 0x7f2ed413f710 2024-03-25T19:43:15.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.379+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726953095 0 0) 0x7f2ed4139840 con 0x7f2ed40a98f0 2024-03-25T19:43:15.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.379+0000 7f2eca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.379+0000 7f2eca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.379+0000 7f2eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.379+0000 7f2eca7fc640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:43:15.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.379+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed41d7380 con 0x7f2ed40a98f0 2024-03-25T19:43:15.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.383+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981575175 0 0) 0x7f2ed41d8200 con 0x7f2ed41430c0 2024-03-25T19:43:15.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.383+0000 7f2eca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.383+0000 7f2eca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.383+0000 7f2eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.383+0000 7f2eca7fc640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:43:15.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.383+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed4139840 con 0x7f2ed41430c0 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.395+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1245968401 0 0) 0x7f2ed41d7380 con 0x7f2ed40a98f0 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.395+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ed41d8200 con 0x7f2ed40a98f0 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.395+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2ed0001b00 con 0x7f2ed40a98f0 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.395+0000 7f2eca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.395+0000 7f2eca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.395+0000 7f2eca7fc640 10 monclient(hunting): dump: 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:15.395 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.395+0000 7f2eca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.397+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064825494 0 0) 0x7f2eb8004eb0 con 0x7f2ed413f710 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.397+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ed41d7380 con 0x7f2ed413f710 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.397+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2ec4001a10 con 0x7f2ed413f710 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.397+0000 7f2eca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.397+0000 7f2eca7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.397+0000 7f2eca7fc640 10 monclient(hunting): dump: 2024-03-25T19:43:15.396 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:15.397 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.402+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3344141769 0 0) 0x7f2ed4139840 con 0x7f2ed41430c0 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.402+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eb8004eb0 con 0x7f2ed41430c0 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.402+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2ecc001a90 con 0x7f2ed41430c0 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.402+0000 7f2eca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.402+0000 7f2eca7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.402+0000 7f2eca7fc640 10 monclient(hunting): dump: 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.402 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.403 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.403 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:15.403 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:15.403 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:15.403 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:15.403 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 270311771 0 0) 0x7f2ed41d7380 con 0x7f2ed413f710 2024-03-25T19:43:15.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 >> v1:172.21.15.18:6790/0 conn(0x7f2ed41430c0 legacy=0x7f2ed41daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 >> v1:172.21.15.53:6789/0 conn(0x7f2ed40a98f0 legacy=0x7f2ed41d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 1 monclient: found mon.a 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed41d75d0 con 0x7f2ed413f710 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2eca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 5 monclient: authenticate success, global_id 4335 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 10 monclient: _renew_subs 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed41db6a0 con 0x7f2ed413f710 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 10 monclient: _renew_subs 2024-03-25T19:43:15.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:15.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed4137910 con 0x7f2ed413f710 2024-03-25T19:43:15.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.422+0000 7f2edc196640 1 librados: init done 2024-03-25T19:43:15.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.423+0000 7f2edc196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.423+0000 7f2edc196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.423+0000 7f2edc196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:15.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.423+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed40627b0 con 0x7f2ed413f710 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.439+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ec4001c50 con 0x7f2ed413f710 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.439+0000 7f2eca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.439+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2ec4003a60 con 0x7f2ed413f710 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.439+0000 7f2eca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.439+0000 7f2eca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.439+0000 7f2eca7fc640 10 monclient: dump: 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:15.439 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:15.440 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:15.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.439+0000 7f2ecbfff640 4 set_mon_vals no callback set 2024-03-25T19:43:15.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.734+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f2ec4003bc0 con 0x7f2ed413f710 2024-03-25T19:43:15.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.735+0000 7f2ed9f32640 10 monclient: get_auth_request con 0x7f2eb806e160 auth_method 0 2024-03-25T19:43:15.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.735+0000 7f2eca7fc640 10 client.4335.objecter ms_handle_connect 0x7f2eb806e160 2024-03-25T19:43:15.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.743+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f2ec4054870 con 0x7f2ed413f710 2024-03-25T19:43:15.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.743+0000 7f2eca7fc640 10 client.4335.objecter ms_dispatch 0x7f2ed4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:15.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.743+0000 7f2eca7fc640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:15.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.743+0000 7f2eca7fc640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:15.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.743+0000 7f2eca7fc640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-25T19:43:15.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.743+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ec4050380 con 0x7f2ed413f710 2024-03-25T19:43:15.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.743+0000 7f2eca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.743+0000 7f2eca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:15.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.988+0000 7f2edc196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:15.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.988+0000 7f2edc196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:15.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.988+0000 7f2edc196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:15.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:15.988+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ed41d7760 con 0x7f2ed413f710 2024-03-25T19:43:16.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.011+0000 7f2eca7fc640 1 -- 172.21.15.18:0/4072910365 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f2ec40522b0 con 0x7f2ed413f710 2024-03-25T19:43:16.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.011+0000 7f2eca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:16.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.011+0000 7f2eca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:16.011 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:43:16.011 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 10 librados: watch_flush enter 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 10 librados: watch_flush exit 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 10 client.4335.objecter successfully canceled tick 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 >> v1:172.21.15.18:6800/1520000573 conn(0x7f2eb806e160 legacy=0x7f2eb8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 10 monclient: shutdown 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 >> v1:172.21.15.18:6789/0 conn(0x7f2ed413f710 legacy=0x7f2ed41da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:16.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 shutdown_connections 2024-03-25T19:43:16.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 >> 172.21.15.18:0/4072910365 conn(0x7f2ed40b7ea0 msgr2=0x7f2ed4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:16.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 shutdown_connections 2024-03-25T19:43:16.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 1 -- 172.21.15.18:0/4072910365 wait complete. 2024-03-25T19:43:16.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:16.013+0000 7f2edc196640 1 librados: shutdown 2024-03-25T19:43:16.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:22.832 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:22.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.945+0000 7fabe7dca640 10 monclient: get_monmap_and_config 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.945+0000 7fabe7dca640 10 monclient: build_initial_monmap 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.945+0000 7fabe7dca640 10 monclient: monmap: 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:43:22.946240+0000 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:43:22.946240+0000 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:43:22.945 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:22.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 1 Processor -- start 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 1 -- start start 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 10 monclient: init 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 10 monclient(hunting): picked mon.noname-b con 0x7fabe00a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 10 monclient(hunting): picked mon.noname-a con 0x7fabe013f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 10 monclient(hunting): picked mon.noname-c con 0x7fabe01430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe0142520 con 0x7fabe013f710 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00b28a0 con 0x7fabe01430c0 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00b2a30 con 0x7fabe00a98f0 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:22.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.946+0000 7fabe7dca640 10 monclient(hunting): authenticate will time out at 5030.073575s 2024-03-25T19:43:22.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.957+0000 7fabe5b66640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fabe00a98f0 0x7fabe00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:36528/0 (socket says 172.21.15.18:36528) 2024-03-25T19:43:22.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.957+0000 7fabe5b66640 1 -- 172.21.15.18:0/3955059398 learned_addr learned my addr 172.21.15.18:0/3955059398 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.962+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769124347 0 0) 0x7fabe00b2a30 con 0x7fabe00a98f0 2024-03-25T19:43:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.962+0000 7fabe4b64640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.962+0000 7fabe4b64640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.962+0000 7fabe4b64640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.962+0000 7fabe4b64640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:43:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.962+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc003520 con 0x7fabe00a98f0 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.975+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791960730 0 0) 0x7fabe00b28a0 con 0x7fabe01430c0 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.975+0000 7fabe4b64640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.975+0000 7fabe4b64640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.975+0000 7fabe4b64640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.975+0000 7fabe4b64640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:43:22.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.975+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe00b2a30 con 0x7fabe01430c0 2024-03-25T19:43:22.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.980+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395876440 0 0) 0x7fabe0142520 con 0x7fabe013f710 2024-03-25T19:43:22.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.980+0000 7fabe4b64640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.980+0000 7fabe4b64640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.980+0000 7fabe4b64640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.981+0000 7fabe4b64640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:43:22.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.981+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe00b28a0 con 0x7fabe013f710 2024-03-25T19:43:22.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291466376 0 0) 0x7fabcc003520 con 0x7fabe00a98f0 2024-03-25T19:43:22.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 >> v1:172.21.15.53:6789/0 conn(0x7fabe01430c0 legacy=0x7fabe014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 >> v1:172.21.15.18:6789/0 conn(0x7fabe013f710 legacy=0x7fabe01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 10 monclient: _finish_hunting 0 2024-03-25T19:43:22.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 1 monclient: found mon.noname-b 2024-03-25T19:43:22.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:22.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:43:22.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00b44c0 con 0x7fabe00a98f0 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 10 monclient: _finish_auth 0 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 10 monclient: _check_auth_tickets 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fabdc001690 con 0x7fabe00a98f0 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe7dca640 5 monclient: authenticate success, global_id 4313 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe7dca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe4b64640 10 monclient: dump: 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:22.985 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.986 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:22.986 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:22.986 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:22.986 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:22.986 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:22.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:22.984+0000 7fabe7dca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:23.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabdc0017f0 con 0x7fabe00a98f0 2024-03-25T19:43:23.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe4b64640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:23.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe4b64640 1 -- 172.21.15.18:0/3955059398 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fabdc001c00 con 0x7fabe00a98f0 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe7dca640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe7dca640 4 set_mon_vals no callback set 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe4b64640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe4b64640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe4b64640 10 monclient: dump: 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:23.007 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:23.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe7dca640 10 monclient: shutdown 2024-03-25T19:43:23.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe7dca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:23.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe7dca640 1 -- 172.21.15.18:0/3955059398 >> v1:172.21.15.18:6790/0 conn(0x7fabe00a98f0 legacy=0x7fabe00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe7dca640 1 -- 172.21.15.18:0/3955059398 shutdown_connections 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.007+0000 7fabe7dca640 1 -- 172.21.15.18:0/3955059398 >> 172.21.15.18:0/3955059398 conn(0x7fabe00b7ea0 msgr2=0x7fabe00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.008+0000 7fabe7dca640 1 -- 172.21.15.18:0/3955059398 shutdown_connections 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.008+0000 7fabe7dca640 1 -- 172.21.15.18:0/3955059398 wait complete. 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.008+0000 7fabe7dca640 10 monclient: build_initial_monmap 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.008+0000 7fabe7dca640 10 monclient: monmap: 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:43:23.010 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 librados: starting msgr at 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 librados: starting objecter 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 Processor -- start 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 -- start start 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 librados: setting wanted keys 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 librados: calling monclient init 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 10 monclient: init 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 10 monclient(hunting): picked mon.noname-b con 0x7fabe00a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 10 monclient(hunting): picked mon.noname-a con 0x7fabe013f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 10 monclient(hunting): picked mon.noname-c con 0x7fabe01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00abd30 con 0x7fabe013f710 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00abfa0 con 0x7fabe00a98f0 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe01d77a0 con 0x7fabe01430c0 2024-03-25T19:43:23.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:23.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.009+0000 7fabe7dca640 10 monclient(hunting): authenticate will time out at 5030.136660s 2024-03-25T19:43:23.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.016+0000 7fabe5365640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fabe013f710 0x7fabe01d4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:40416/0 (socket says 172.21.15.18:40416) 2024-03-25T19:43:23.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.016+0000 7fabe5365640 1 -- 172.21.15.18:0/3086034725 learned_addr learned my addr 172.21.15.18:0/3086034725 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:23.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.016+0000 7fabd67fc640 10 client.?.objecter ms_handle_connect 0x7fabe013f710 2024-03-25T19:43:23.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.016+0000 7fabd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.019+0000 7fabd67fc640 10 client.?.objecter ms_handle_connect 0x7fabe01430c0 2024-03-25T19:43:23.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.019+0000 7fabd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:23.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.023+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989004672 0 0) 0x7fabe00abd30 con 0x7fabe013f710 2024-03-25T19:43:23.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.023+0000 7fabd67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:23.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.023+0000 7fabd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:23.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.023+0000 7fabd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:23.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.023+0000 7fabd67fc640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:43:23.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.023+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc4004eb0 con 0x7fabe013f710 2024-03-25T19:43:23.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.030+0000 7fabd67fc640 10 client.?.objecter ms_handle_connect 0x7fabe00a98f0 2024-03-25T19:43:23.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.030+0000 7fabd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:23.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.038+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762057632 0 0) 0x7fabe01d77a0 con 0x7fabe01430c0 2024-03-25T19:43:23.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.039+0000 7fabd67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:23.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.039+0000 7fabd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:23.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.039+0000 7fabd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:23.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.039+0000 7fabd67fc640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:43:23.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.039+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe00abd30 con 0x7fabe01430c0 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.043+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441274915 0 0) 0x7fabc4004eb0 con 0x7fabe013f710 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.043+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe01d77a0 con 0x7fabe013f710 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.043+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fabd0001a10 con 0x7fabe013f710 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.043+0000 7fabd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.043+0000 7fabd67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.043+0000 7fabd67fc640 10 monclient(hunting): dump: 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.043 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:23.044 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:23.044 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:23.044 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:23.044 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:23.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.043+0000 7fabd67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:23.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.049+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300749998 0 0) 0x7fabe00abfa0 con 0x7fabe00a98f0 2024-03-25T19:43:23.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.049+0000 7fabd67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:23.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.049+0000 7fabd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:23.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.049+0000 7fabd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:23.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.049+0000 7fabd67fc640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:43:23.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.049+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc4004eb0 con 0x7fabe00a98f0 2024-03-25T19:43:23.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3743227486 0 0) 0x7fabe01d77a0 con 0x7fabe013f710 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 >> v1:172.21.15.18:6790/0 conn(0x7fabe01430c0 legacy=0x7fabe01385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 >> v1:172.21.15.53:6789/0 conn(0x7fabe00a98f0 legacy=0x7fabe01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 1 monclient: found mon.a 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe01d83d0 con 0x7fabe013f710 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:23.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 5 monclient: authenticate success, global_id 4341 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 10 monclient: _renew_subs 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe01d41e0 con 0x7fabe013f710 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 10 monclient: _renew_subs 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe01d5110 con 0x7fabe013f710 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.051+0000 7fabe7dca640 1 librados: init done 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.052+0000 7fabe7dca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.052+0000 7fabe7dca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.052+0000 7fabe7dca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:23.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.052+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe01d7930 con 0x7fabe013f710 2024-03-25T19:43:23.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.263+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd0001c70 con 0x7fabe013f710 2024-03-25T19:43:23.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.263+0000 7fabd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:23.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.263+0000 7fabd7fff640 4 set_mon_vals no callback set 2024-03-25T19:43:23.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.376+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fabd0003a60 con 0x7fabe013f710 2024-03-25T19:43:23.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.376+0000 7fabd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.376+0000 7fabd67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.376+0000 7fabd67fc640 10 monclient: dump: 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:23.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.377+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fabd0003c40 con 0x7fabe013f710 2024-03-25T19:43:23.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.378+0000 7fabe5b66640 10 monclient: get_auth_request con 0x7fabc406de00 auth_method 0 2024-03-25T19:43:23.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.378+0000 7fabd67fc640 10 client.4341.objecter ms_handle_connect 0x7fabc406de00 2024-03-25T19:43:23.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.387+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fabd00548f0 con 0x7fabe013f710 2024-03-25T19:43:23.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.387+0000 7fabd67fc640 10 client.4341.objecter ms_dispatch 0x7fabe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:23.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.387+0000 7fabd67fc640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:23.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.387+0000 7fabd67fc640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:23.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.387+0000 7fabd67fc640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-25T19:43:23.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.387+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabd00b18b0 con 0x7fabe013f710 2024-03-25T19:43:23.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.387+0000 7fabd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:23.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.387+0000 7fabd67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:23.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.645+0000 7fabe7dca640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.645+0000 7fabe7dca640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.645+0000 7fabe7dca640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:23.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.645+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabe0136a20 con 0x7fabe013f710 2024-03-25T19:43:23.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.672+0000 7fabd67fc640 1 -- 172.21.15.18:0/3086034725 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fabd0053fc0 con 0x7fabe013f710 2024-03-25T19:43:23.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.672+0000 7fabd67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.672+0000 7fabd67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:23.672 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:43:23.672 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:23.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 10 librados: watch_flush enter 2024-03-25T19:43:23.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 10 librados: watch_flush exit 2024-03-25T19:43:23.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-25T19:43:23.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 10 client.4341.objecter successfully canceled tick 2024-03-25T19:43:23.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 >> v1:172.21.15.18:6800/1520000573 conn(0x7fabc406de00 legacy=0x7fabc40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 10 monclient: shutdown 2024-03-25T19:43:23.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:23.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 >> v1:172.21.15.18:6789/0 conn(0x7fabe013f710 legacy=0x7fabe01d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 shutdown_connections 2024-03-25T19:43:23.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 >> 172.21.15.18:0/3086034725 conn(0x7fabe00b7ea0 msgr2=0x7fabe013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:23.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 shutdown_connections 2024-03-25T19:43:23.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 1 -- 172.21.15.18:0/3086034725 wait complete. 2024-03-25T19:43:23.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:23.674+0000 7fabe7dca640 1 librados: shutdown 2024-03-25T19:43:23.685 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:27.112 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:54.642 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:27.173 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:54.704 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:27.205 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:54.736 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:27.218 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:54.749 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:27.218 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:54.715 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:27.227 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:54.723 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:27.231 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:54.762 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:27.253 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:02:54.749 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:27.258 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:54.789 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:30.492 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:30.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.608+0000 7fc4209d6640 10 monclient: get_monmap_and_config 2024-03-25T19:43:30.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.608+0000 7fc4209d6640 10 monclient: build_initial_monmap 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.608+0000 7fc4209d6640 10 monclient: monmap: 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:43:30.609700+0000 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:43:30.609700+0000 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:43:30.609 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:43:30.610 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.609+0000 7fc4209d6640 1 Processor -- start 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.609+0000 7fc4209d6640 1 -- start start 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.609+0000 7fc4209d6640 10 monclient: init 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c142520 con 0x7fc41c1430c0 2024-03-25T19:43:30.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0b28a0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0b2a30 con 0x7fc41c13f710 2024-03-25T19:43:30.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:30.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.610+0000 7fc4209d6640 10 monclient(hunting): authenticate will time out at 5037.737598s 2024-03-25T19:43:30.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.618+0000 7fc41a59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc41c0a98f0 0x7fc41c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:53712/0 (socket says 172.21.15.18:53712) 2024-03-25T19:43:30.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.618+0000 7fc41a59c640 1 -- 172.21.15.18:0/1114647053 learned_addr learned my addr 172.21.15.18:0/1114647053 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.635+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945025731 0 0) 0x7fc41c0b28a0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.635+0000 7fc41959a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.635+0000 7fc41959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.635+0000 7fc41959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.635+0000 7fc41959a640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:43:30.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.635+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc404003520 con 0x7fc41c0a98f0 2024-03-25T19:43:30.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.636+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265648233 0 0) 0x7fc41c0b2a30 con 0x7fc41c13f710 2024-03-25T19:43:30.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.636+0000 7fc41959a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.636+0000 7fc41959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.636+0000 7fc41959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.636+0000 7fc41959a640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:43:30.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.636+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c0b28a0 con 0x7fc41c13f710 2024-03-25T19:43:30.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.643+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877500055 0 0) 0x7fc41c142520 con 0x7fc41c1430c0 2024-03-25T19:43:30.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.643+0000 7fc41959a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.643+0000 7fc41959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.643+0000 7fc41959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.643+0000 7fc41959a640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:43:30.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.643+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c0b2a30 con 0x7fc41c1430c0 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.650+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355641530 0 0) 0x7fc404003520 con 0x7fc41c0a98f0 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 >> v1:172.21.15.18:6790/0 conn(0x7fc41c13f710 legacy=0x7fc41c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 >> v1:172.21.15.18:6789/0 conn(0x7fc41c1430c0 legacy=0x7fc41c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 10 monclient: _finish_hunting 0 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 1 monclient: found mon.noname-c 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0b44c0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 10 monclient: _finish_auth 0 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 10 monclient: _check_auth_tickets 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:30.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc40c001690 con 0x7fc41c0a98f0 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc41959a640 10 monclient: dump: 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc4209d6640 5 monclient: authenticate success, global_id 4318 2024-03-25T19:43:30.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.651+0000 7fc4209d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:30.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.659+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc40c0017f0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.659+0000 7fc41959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:30.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.659+0000 7fc41959a640 1 -- 172.21.15.18:0/1114647053 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc40c001c00 con 0x7fc41c0a98f0 2024-03-25T19:43:30.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.659+0000 7fc41959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.659+0000 7fc41959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:30.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.659+0000 7fc41959a640 10 monclient: dump: 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.660+0000 7fc4209d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:30.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.660+0000 7fc4209d6640 4 set_mon_vals no callback set 2024-03-25T19:43:30.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.660+0000 7fc4209d6640 10 monclient: shutdown 2024-03-25T19:43:30.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.660+0000 7fc4209d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.660+0000 7fc4209d6640 1 -- 172.21.15.18:0/1114647053 >> v1:172.21.15.53:6789/0 conn(0x7fc41c0a98f0 legacy=0x7fc41c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.660+0000 7fc4209d6640 1 -- 172.21.15.18:0/1114647053 shutdown_connections 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.660+0000 7fc4209d6640 1 -- 172.21.15.18:0/1114647053 >> 172.21.15.18:0/1114647053 conn(0x7fc41c0b7ea0 msgr2=0x7fc41c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.661+0000 7fc4209d6640 1 -- 172.21.15.18:0/1114647053 shutdown_connections 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.661+0000 7fc4209d6640 1 -- 172.21.15.18:0/1114647053 wait complete. 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.661+0000 7fc4209d6640 10 monclient: build_initial_monmap 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.661+0000 7fc4209d6640 10 monclient: monmap: 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:30.663 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 1 librados: starting msgr at 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 1 librados: starting objecter 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 1 Processor -- start 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 1 -- start start 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 1 librados: setting wanted keys 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 1 librados: calling monclient init 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 10 monclient: init 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:30.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c138250 con 0x7fc41c1430c0 2024-03-25T19:43:30.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.662+0000 7fc4209d6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c137d50 con 0x7fc41c13f710 2024-03-25T19:43:30.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.663+0000 7fc4209d6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c137ee0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.663+0000 7fc4209d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:30.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.663+0000 7fc4209d6640 10 monclient(hunting): authenticate will time out at 5037.789749s 2024-03-25T19:43:30.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.674+0000 7fc41a59c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc41c0a98f0 0x7fc41c137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46384/0 (socket says 172.21.15.18:46384) 2024-03-25T19:43:30.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.674+0000 7fc41a59c640 1 -- 172.21.15.18:0/2275808816 learned_addr learned my addr 172.21.15.18:0/2275808816 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:30.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.676+0000 7fc40affd640 10 client.?.objecter ms_handle_connect 0x7fc41c0a98f0 2024-03-25T19:43:30.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.676+0000 7fc40affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.677+0000 7fc40affd640 10 client.?.objecter ms_handle_connect 0x7fc41c1430c0 2024-03-25T19:43:30.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.678+0000 7fc40affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.678+0000 7fc40affd640 10 client.?.objecter ms_handle_connect 0x7fc41c13f710 2024-03-25T19:43:30.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.678+0000 7fc40affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.685+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324237340 0 0) 0x7fc41c137ee0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.685+0000 7fc40affd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.685+0000 7fc40affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.685+0000 7fc40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.685+0000 7fc40affd640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:43:30.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.685+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3fc004eb0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.689+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825911926 0 0) 0x7fc41c137d50 con 0x7fc41c13f710 2024-03-25T19:43:30.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.689+0000 7fc40affd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.689+0000 7fc40affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.689+0000 7fc40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.689+0000 7fc40affd640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:43:30.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.689+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c137ee0 con 0x7fc41c13f710 2024-03-25T19:43:30.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.692+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603073578 0 0) 0x7fc41c138250 con 0x7fc41c1430c0 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.692+0000 7fc40affd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.692+0000 7fc40affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.692+0000 7fc40affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.692+0000 7fc40affd640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.692+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc41c137d50 con 0x7fc41c1430c0 2024-03-25T19:43:30.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.711+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1523483276 0 0) 0x7fc3fc004eb0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.711+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc41c138250 con 0x7fc41c0a98f0 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.711+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc40c001bc0 con 0x7fc41c0a98f0 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.711+0000 7fc40affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.711+0000 7fc40affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.711+0000 7fc40affd640 10 monclient(hunting): dump: 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:30.711 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:30.712 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:30.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.711+0000 7fc40affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:30.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.713+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1138132670 0 0) 0x7fc41c137ee0 con 0x7fc41c13f710 2024-03-25T19:43:30.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.713+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3fc004eb0 con 0x7fc41c13f710 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.713+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc410001970 con 0x7fc41c13f710 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.713+0000 7fc40affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.713+0000 7fc40affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.713+0000 7fc40affd640 10 monclient(hunting): dump: 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:30.713 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.716+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2699238121 0 0) 0x7fc41c137d50 con 0x7fc41c1430c0 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.716+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc41c137ee0 con 0x7fc41c1430c0 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.716+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc414001bd0 con 0x7fc41c1430c0 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.716+0000 7fc40affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.717+0000 7fc40affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.717+0000 7fc40affd640 10 monclient(hunting): dump: 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:30.716 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.717 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:30.717 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.717 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:30.717 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:30.717 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:30.717 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:30.717 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:30.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3154942241 0 0) 0x7fc3fc004eb0 con 0x7fc41c13f710 2024-03-25T19:43:30.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 >> v1:172.21.15.18:6790/0 conn(0x7fc41c0a98f0 legacy=0x7fc41c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 >> v1:172.21.15.18:6789/0 conn(0x7fc41c1430c0 legacy=0x7fc41c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 1 monclient: found mon.b 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0b7730 con 0x7fc41c13f710 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 10 monclient: _finish_auth 0 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc40affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 10 monclient: _renew_subs 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c0b7920 con 0x7fc41c13f710 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:30.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 10 monclient: _renew_subs 2024-03-25T19:43:30.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:30.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c1e33d0 con 0x7fc41c13f710 2024-03-25T19:43:30.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.730+0000 7fc4209d6640 1 librados: init done 2024-03-25T19:43:30.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.732+0000 7fc4209d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.732+0000 7fc4209d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.732+0000 7fc4209d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:30.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:30.732+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c0627b0 con 0x7fc41c13f710 2024-03-25T19:43:31.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.019+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4100035f0 con 0x7fc41c13f710 2024-03-25T19:43:31.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.019+0000 7fc40affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:31.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.019+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc410003900 con 0x7fc41c13f710 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.019+0000 7fc40affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.020+0000 7fc40affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.020+0000 7fc40affd640 10 monclient: dump: 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:31.020 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:31.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.020+0000 7fc418d99640 4 set_mon_vals no callback set 2024-03-25T19:43:31.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.020+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fc4100198d0 con 0x7fc41c13f710 2024-03-25T19:43:31.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.021+0000 7fc41a59c640 10 monclient: get_auth_request con 0x7fc3fc06e0c0 auth_method 0 2024-03-25T19:43:31.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.021+0000 7fc40affd640 10 client.4321.objecter ms_handle_connect 0x7fc3fc06e0c0 2024-03-25T19:43:31.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.026+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fc4100535b0 con 0x7fc41c13f710 2024-03-25T19:43:31.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.026+0000 7fc40affd640 10 client.4321.objecter ms_dispatch 0x7fc41c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:31.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.026+0000 7fc40affd640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:31.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.026+0000 7fc40affd640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:31.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.026+0000 7fc40affd640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T19:43:31.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.027+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc41004f0c0 con 0x7fc41c13f710 2024-03-25T19:43:31.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.027+0000 7fc40affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:31.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.027+0000 7fc40affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:31.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.268+0000 7fc4209d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.268+0000 7fc4209d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.268+0000 7fc4209d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:31.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.268+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc41c13df50 con 0x7fc41c13f710 2024-03-25T19:43:31.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.312+0000 7fc40affd640 1 -- 172.21.15.18:0/2275808816 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fc410050ff0 con 0x7fc41c13f710 2024-03-25T19:43:31.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.312+0000 7fc40affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.312+0000 7fc40affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:31.312 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:43:31.312 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:31.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.314+0000 7fc4209d6640 10 librados: watch_flush enter 2024-03-25T19:43:31.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.314+0000 7fc4209d6640 10 librados: watch_flush exit 2024-03-25T19:43:31.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.314+0000 7fc4209d6640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.314+0000 7fc4209d6640 10 client.4321.objecter successfully canceled tick 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.314+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 >> v1:172.21.15.18:6800/1520000573 conn(0x7fc3fc06e0c0 legacy=0x7fc3fc070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.315+0000 7fc4209d6640 10 monclient: shutdown 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.315+0000 7fc4209d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.315+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 >> v1:172.21.15.53:6789/0 conn(0x7fc41c13f710 legacy=0x7fc41c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.315+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 shutdown_connections 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.315+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 >> 172.21.15.18:0/2275808816 conn(0x7fc41c0b7ea0 msgr2=0x7fc41c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.315+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 shutdown_connections 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.315+0000 7fc4209d6640 1 -- 172.21.15.18:0/2275808816 wait complete. 2024-03-25T19:43:31.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:31.315+0000 7fc4209d6640 1 librados: shutdown 2024-03-25T19:43:31.326 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:38.133 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:38.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.251+0000 7f5191c84640 10 monclient: get_monmap_and_config 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.251+0000 7f5191c84640 10 monclient: build_initial_monmap 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.251+0000 7f5191c84640 10 monclient: monmap: 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:43:38.252168+0000 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:43:38.252168+0000 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:43:38.251 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:38.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.252+0000 7f5191c84640 1 Processor -- start 2024-03-25T19:43:38.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.252+0000 7f5191c84640 1 -- start start 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.252+0000 7f5191c84640 10 monclient: init 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 10 monclient(hunting): picked mon.noname-c con 0x7f518c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 10 monclient(hunting): picked mon.noname-a con 0x7f518c13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 10 monclient(hunting): picked mon.noname-b con 0x7f518c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c142520 con 0x7f518c13f710 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c0b28a0 con 0x7f518c0a98f0 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c0b2a30 con 0x7f518c1430c0 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:38.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.253+0000 7f5191c84640 10 monclient(hunting): authenticate will time out at 5045.380125s 2024-03-25T19:43:38.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.257+0000 7f518affd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f518c13f710 0x7f518c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60016/0 (socket says 172.21.15.18:60016) 2024-03-25T19:43:38.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.257+0000 7f518affd640 1 -- 172.21.15.18:0/1731856487 learned_addr learned my addr 172.21.15.18:0/1731856487 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:38.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.272+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60367447 0 0) 0x7f518c142520 con 0x7f518c13f710 2024-03-25T19:43:38.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.272+0000 7f518a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.272+0000 7f518a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.272+0000 7f518a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.272+0000 7f518a7fc640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:43:38.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.272+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5174003520 con 0x7f518c13f710 2024-03-25T19:43:38.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3881665098 0 0) 0x7f5174003520 con 0x7f518c13f710 2024-03-25T19:43:38.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 >> v1:172.21.15.18:6790/0 conn(0x7f518c1430c0 legacy=0x7f518c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:43:38.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 >> v1:172.21.15.53:6789/0 conn(0x7f518c0a98f0 legacy=0x7f518c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c0b44c0 con 0x7f518c13f710 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5180001690 con 0x7f518c13f710 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f5191c84640 5 monclient: authenticate success, global_id 4350 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f5191c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f518a7fc640 10 monclient: dump: 2024-03-25T19:43:38.283 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:38.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.282+0000 7f5191c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:38.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.291+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5180001860 con 0x7f518c13f710 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.291+0000 7f518a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.291+0000 7f518a7fc640 1 -- 172.21.15.18:0/1731856487 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5180001cb0 con 0x7f518c13f710 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.291+0000 7f518a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.291+0000 7f518a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.291+0000 7f518a7fc640 10 monclient: dump: 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:38.291 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:38.292 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:38.292 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:38.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.291+0000 7f5191c84640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:38.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.291+0000 7f5191c84640 4 set_mon_vals no callback set 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 10 monclient: shutdown 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 1 -- 172.21.15.18:0/1731856487 >> v1:172.21.15.18:6789/0 conn(0x7f518c13f710 legacy=0x7f518c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 1 -- 172.21.15.18:0/1731856487 shutdown_connections 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 1 -- 172.21.15.18:0/1731856487 >> 172.21.15.18:0/1731856487 conn(0x7f518c0b7ea0 msgr2=0x7f518c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 1 -- 172.21.15.18:0/1731856487 shutdown_connections 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 1 -- 172.21.15.18:0/1731856487 wait complete. 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 10 monclient: build_initial_monmap 2024-03-25T19:43:38.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.292+0000 7f5191c84640 10 monclient: monmap: 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.293+0000 7f5191c84640 1 librados: starting msgr at 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.293+0000 7f5191c84640 1 librados: starting objecter 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.293+0000 7f5191c84640 1 Processor -- start 2024-03-25T19:43:38.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.293+0000 7f5191c84640 1 -- start start 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.293+0000 7f5191c84640 1 librados: setting wanted keys 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.293+0000 7f5191c84640 1 librados: calling monclient init 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.293+0000 7f5191c84640 10 monclient: init 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 10 monclient(hunting): picked mon.noname-b con 0x7f518c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 10 monclient(hunting): picked mon.noname-c con 0x7f518c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 10 monclient(hunting): picked mon.noname-a con 0x7f518c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c1369a0 con 0x7f518c1430c0 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c136ef0 con 0x7f518c0a98f0 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c136c10 con 0x7f518c13f710 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:38.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.294+0000 7f5191c84640 10 monclient(hunting): authenticate will time out at 5045.421064s 2024-03-25T19:43:38.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.298+0000 7f518bfff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f518c1430c0 0x7f518c1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60044/0 (socket says 172.21.15.18:60044) 2024-03-25T19:43:38.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.298+0000 7f518bfff640 1 -- 172.21.15.18:0/2741732890 learned_addr learned my addr 172.21.15.18:0/2741732890 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:38.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.300+0000 7f5173fff640 10 client.?.objecter ms_handle_connect 0x7f518c1430c0 2024-03-25T19:43:38.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.300+0000 7f5173fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.306+0000 7f5173fff640 10 client.?.objecter ms_handle_connect 0x7f518c13f710 2024-03-25T19:43:38.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.306+0000 7f5173fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.309+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533508956 0 0) 0x7f518c1369a0 con 0x7f518c1430c0 2024-03-25T19:43:38.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.309+0000 7f5173fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.309+0000 7f5173fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.309+0000 7f5173fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.309+0000 7f5173fff640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:43:38.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.309+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5178004eb0 con 0x7f518c1430c0 2024-03-25T19:43:38.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.315+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777620692 0 0) 0x7f518c136c10 con 0x7f518c13f710 2024-03-25T19:43:38.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.315+0000 7f5173fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.315+0000 7f5173fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.315+0000 7f5173fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.315+0000 7f5173fff640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:43:38.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.315+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c1369a0 con 0x7f518c13f710 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.320+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2282373545 0 0) 0x7f5178004eb0 con 0x7f518c1430c0 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.320+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f518c136c10 con 0x7f518c1430c0 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.320+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f51840042e0 con 0x7f518c1430c0 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.320+0000 7f5173fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.320+0000 7f5173fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.320+0000 7f5173fff640 10 monclient(hunting): dump: 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.320 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:38.321 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:38.321 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:38.321 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:38.321 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:38.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.320+0000 7f5173fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:38.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.327+0000 7f5173fff640 10 client.?.objecter ms_handle_connect 0x7f518c0a98f0 2024-03-25T19:43:38.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.327+0000 7f5173fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.331+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 542898962 0 0) 0x7f518c1369a0 con 0x7f518c13f710 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.331+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5178004eb0 con 0x7f518c13f710 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.331+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5180001b70 con 0x7f518c13f710 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.331+0000 7f5173fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.331+0000 7f5173fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.331+0000 7f5173fff640 10 monclient(hunting): dump: 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:38.331 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.332 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.332 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.332 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:38.332 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:38.332 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:38.332 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:38.332 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.341+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325570207 0 0) 0x7f518c136ef0 con 0x7f518c0a98f0 2024-03-25T19:43:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.341+0000 7f5173fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.341+0000 7f5173fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.341+0000 7f5173fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.341+0000 7f5173fff640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:43:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.341+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f518c1369a0 con 0x7f518c0a98f0 2024-03-25T19:43:38.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1426816261 0 0) 0x7f518c136c10 con 0x7f518c1430c0 2024-03-25T19:43:38.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 >> v1:172.21.15.18:6790/0 conn(0x7f518c13f710 legacy=0x7f518c1379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 >> v1:172.21.15.53:6789/0 conn(0x7f518c0a98f0 legacy=0x7f518c13b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:38.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 1 monclient: found mon.a 2024-03-25T19:43:38.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c1371d0 con 0x7f518c1430c0 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5173fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 5 monclient: authenticate success, global_id 4353 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 10 monclient: _renew_subs 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c1361b0 con 0x7f518c1430c0 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 10 monclient: _renew_subs 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:38.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c135300 con 0x7f518c1430c0 2024-03-25T19:43:38.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.345+0000 7f5191c84640 1 librados: init done 2024-03-25T19:43:38.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.346+0000 7f5191c84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.346+0000 7f5191c84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.346+0000 7f5191c84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:38.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.346+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c1376f0 con 0x7f518c1430c0 2024-03-25T19:43:38.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.695+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51840047f0 con 0x7f518c1430c0 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.695+0000 7f5173fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.695+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5184001250 con 0x7f518c1430c0 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.695+0000 7f5173fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.695+0000 7f5173fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.695+0000 7f5173fff640 10 monclient: dump: 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:38.696 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:38.697 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:38.697 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:38.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.695+0000 7f5189ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:38.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.696+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f5184001470 con 0x7f518c1430c0 2024-03-25T19:43:38.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.697+0000 7f518b7fe640 10 monclient: get_auth_request con 0x7f517806df00 auth_method 0 2024-03-25T19:43:38.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.697+0000 7f5173fff640 10 client.4353.objecter ms_handle_connect 0x7f517806df00 2024-03-25T19:43:38.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.702+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f51840548e0 con 0x7f518c1430c0 2024-03-25T19:43:38.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.702+0000 7f5173fff640 10 client.4353.objecter ms_dispatch 0x7f518c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:38.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.702+0000 7f5173fff640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:38.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.702+0000 7f5173fff640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:38.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.702+0000 7f5173fff640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T19:43:38.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.702+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5184050220 con 0x7f518c1430c0 2024-03-25T19:43:38.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.702+0000 7f5173fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.702+0000 7f5173fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:38.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.954+0000 7f5191c84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.954+0000 7f5191c84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.954+0000 7f5191c84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:38.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.954+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f518c0aa500 con 0x7f518c1430c0 2024-03-25T19:43:38.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.987+0000 7f5173fff640 1 -- 172.21.15.18:0/2741732890 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f5184052320 con 0x7f518c1430c0 2024-03-25T19:43:38.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.987+0000 7f5173fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.987+0000 7f5173fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:38.988 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:43:38.988 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 10 librados: watch_flush enter 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 10 librados: watch_flush exit 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 10 client.4353.objecter successfully canceled tick 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 >> v1:172.21.15.18:6800/1520000573 conn(0x7f517806df00 legacy=0x7f51780703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 10 monclient: shutdown 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 >> v1:172.21.15.18:6789/0 conn(0x7f518c1430c0 legacy=0x7f518c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 shutdown_connections 2024-03-25T19:43:38.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 >> 172.21.15.18:0/2741732890 conn(0x7f518c0b7ea0 msgr2=0x7f518c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:38.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 shutdown_connections 2024-03-25T19:43:38.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 1 -- 172.21.15.18:0/2741732890 wait complete. 2024-03-25T19:43:38.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:38.990+0000 7f5191c84640 1 librados: shutdown 2024-03-25T19:43:39.002 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:45.808 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:45.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.917+0000 7f9fac8f4640 10 monclient: get_monmap_and_config 2024-03-25T19:43:45.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.917+0000 7f9fac8f4640 10 monclient: build_initial_monmap 2024-03-25T19:43:45.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.917+0000 7f9fac8f4640 10 monclient: monmap: 2024-03-25T19:43:45.917 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:45.917 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:45.918 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:43:45.918885+0000 2024-03-25T19:43:45.918 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:43:45.918885+0000 2024-03-25T19:43:45.918 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:45.918 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:45.918 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:45.918 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:43:45.918 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:43:45.918 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:45.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.917+0000 7f9fac8f4640 1 Processor -- start 2024-03-25T19:43:45.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.917+0000 7f9fac8f4640 1 -- start start 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.917+0000 7f9fac8f4640 10 monclient: init 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa80b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa80b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa814be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80af940 con 0x7f9fa814be90 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80b0a70 con 0x7f9fa80b4d60 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80b0d50 con 0x7f9fa80b1730 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fac8f4640 10 monclient(hunting): authenticate will time out at 5053.046137s 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fa5d9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9fa80b1730 0x7f9fa80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:43744/0 (socket says 172.21.15.18:43744) 2024-03-25T19:43:45.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.918+0000 7f9fa5d9b640 1 -- 172.21.15.18:0/77504853 learned_addr learned my addr 172.21.15.18:0/77504853 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:45.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.982+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706166529 0 0) 0x7f9fa80af940 con 0x7f9fa814be90 2024-03-25T19:43:45.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.982+0000 7f9fa559a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.982+0000 7f9fa559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.982+0000 7f9fa559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.982+0000 7f9fa559a640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:43:45.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:45.982+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f90003520 con 0x7f9fa814be90 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1367103742 0 0) 0x7f9f90003520 con 0x7f9fa814be90 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 >> v1:172.21.15.18:6790/0 conn(0x7f9fa80b1730 legacy=0x7f9fa80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 >> v1:172.21.15.53:6789/0 conn(0x7f9fa80b4d60 legacy=0x7f9fa813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 10 monclient: _finish_hunting 0 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 1 monclient: found mon.noname-a 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa80b1030 con 0x7f9fa814be90 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 10 monclient: _finish_auth 0 2024-03-25T19:43:46.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 10 monclient: _check_auth_tickets 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9fa0001a10 con 0x7f9fa814be90 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fa559a640 10 monclient: dump: 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:46.005 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:46.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fac8f4640 5 monclient: authenticate success, global_id 4356 2024-03-25T19:43:46.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.003+0000 7f9fac8f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:46.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fa0001b70 con 0x7f9fa814be90 2024-03-25T19:43:46.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fa559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:46.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fa559a640 1 -- 172.21.15.18:0/77504853 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9fa0001f80 con 0x7f9fa814be90 2024-03-25T19:43:46.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fa559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fa559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fa559a640 10 monclient: dump: 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:46.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fac8f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:46.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fac8f4640 4 set_mon_vals no callback set 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fac8f4640 10 monclient: shutdown 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.016+0000 7f9fac8f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.017+0000 7f9fac8f4640 1 -- 172.21.15.18:0/77504853 >> v1:172.21.15.18:6789/0 conn(0x7f9fa814be90 legacy=0x7f9fa814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.017+0000 7f9fac8f4640 1 -- 172.21.15.18:0/77504853 shutdown_connections 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.017+0000 7f9fac8f4640 1 -- 172.21.15.18:0/77504853 >> 172.21.15.18:0/77504853 conn(0x7f9fa80617a0 msgr2=0x7f9fa8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.017+0000 7f9fac8f4640 1 -- 172.21.15.18:0/77504853 shutdown_connections 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.017+0000 7f9fac8f4640 1 -- 172.21.15.18:0/77504853 wait complete. 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 10 monclient: build_initial_monmap 2024-03-25T19:43:46.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 10 monclient: monmap: 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 1 librados: starting msgr at 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 1 librados: starting objecter 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 1 Processor -- start 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 1 -- start start 2024-03-25T19:43:46.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 1 librados: setting wanted keys 2024-03-25T19:43:46.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 1 librados: calling monclient init 2024-03-25T19:43:46.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.018+0000 7f9fac8f4640 10 monclient: init 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa80b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa80b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa814be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa81307c0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8138f90 con 0x7f9fa80b1730 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8132ab0 con 0x7f9fa814be90 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.019+0000 7f9fac8f4640 10 monclient(hunting): authenticate will time out at 5053.147474s 2024-03-25T19:43:46.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.036+0000 7f9fa5d9b640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f9fa80b4d60 0x7f9fa812f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:55974/0 (socket says 172.21.15.18:55974) 2024-03-25T19:43:46.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.036+0000 7f9fa5d9b640 1 -- 172.21.15.18:0/4008687256 learned_addr learned my addr 172.21.15.18:0/4008687256 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:46.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.037+0000 7f9f96ffd640 10 client.?.objecter ms_handle_connect 0x7f9fa80b4d60 2024-03-25T19:43:46.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.037+0000 7f9f96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:46.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.038+0000 7f9f96ffd640 10 client.?.objecter ms_handle_connect 0x7f9fa80b1730 2024-03-25T19:43:46.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.038+0000 7f9f96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:46.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.046+0000 7f9f96ffd640 10 client.?.objecter ms_handle_connect 0x7f9fa814be90 2024-03-25T19:43:46.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.046+0000 7f9f96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:46.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.050+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010464742 0 0) 0x7f9fa81307c0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.050+0000 7f9f96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.050+0000 7f9f96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.050+0000 7f9f96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.050+0000 7f9f96ffd640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:43:46.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.050+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f88004eb0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.114+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517830912 0 0) 0x7f9fa8132ab0 con 0x7f9fa814be90 2024-03-25T19:43:46.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.114+0000 7f9f96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.114+0000 7f9f96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.114+0000 7f9f96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.114+0000 7f9f96ffd640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:43:46.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.114+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa81307c0 con 0x7f9fa814be90 2024-03-25T19:43:46.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.116+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465220463 0 0) 0x7f9fa8138f90 con 0x7f9fa80b1730 2024-03-25T19:43:46.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.116+0000 7f9f96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:46.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.116+0000 7f9f96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:46.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.116+0000 7f9f96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:46.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.116+0000 7f9f96ffd640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:43:46.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.116+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa8132ab0 con 0x7f9fa80b1730 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.122+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 152781018 0 0) 0x7f9f88004eb0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.122+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa8138f90 con 0x7f9fa80b4d60 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.122+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9f9c0018b0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.122+0000 7f9f96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.122+0000 7f9f96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.122+0000 7f9f96ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:46.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.122+0000 7f9f96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:46.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.126+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2766451804 0 0) 0x7f9fa8132ab0 con 0x7f9fa80b1730 2024-03-25T19:43:46.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.126+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f88004eb0 con 0x7f9fa80b1730 2024-03-25T19:43:46.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.126+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9f98001860 con 0x7f9fa80b1730 2024-03-25T19:43:46.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.126+0000 7f9f96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.126+0000 7f9f96ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:46.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.126+0000 7f9f96ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:46.127 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:46.128 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:46.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2860100846 0 0) 0x7f9fa8138f90 con 0x7f9fa80b4d60 2024-03-25T19:43:46.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 >> v1:172.21.15.18:6790/0 conn(0x7f9fa814be90 legacy=0x7f9fa81300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 >> v1:172.21.15.53:6789/0 conn(0x7f9fa80b1730 legacy=0x7f9fa81393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:46.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 1 monclient: found mon.a 2024-03-25T19:43:46.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:46.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:46.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8133a40 con 0x7f9fa80b4d60 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9f96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 5 monclient: authenticate success, global_id 4359 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 10 monclient: _renew_subs 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa81316c0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 10 monclient: _renew_subs 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa8137860 con 0x7f9fa80b4d60 2024-03-25T19:43:46.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.134+0000 7f9fac8f4640 1 librados: init done 2024-03-25T19:43:46.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.135+0000 7f9fac8f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:46.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.135+0000 7f9fac8f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:46.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.135+0000 7f9fac8f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:46.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.135+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8132c40 con 0x7f9fa80b4d60 2024-03-25T19:43:46.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.453+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f9c0047e0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.453+0000 7f9f96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:46.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.453+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9f9c004c30 con 0x7f9fa80b4d60 2024-03-25T19:43:46.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.453+0000 7f9f96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.453+0000 7f9f96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:46.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.453+0000 7f9f96ffd640 10 monclient: dump: 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:46.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.453+0000 7f9fa4d99640 4 set_mon_vals no callback set 2024-03-25T19:43:46.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.454+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f9f9c004e50 con 0x7f9fa80b4d60 2024-03-25T19:43:46.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.455+0000 7f9fa659c640 10 monclient: get_auth_request con 0x7f9f8806df70 auth_method 0 2024-03-25T19:43:46.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.455+0000 7f9f96ffd640 10 client.4359.objecter ms_handle_connect 0x7f9f8806df70 2024-03-25T19:43:46.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.462+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f9f9c054870 con 0x7f9fa80b4d60 2024-03-25T19:43:46.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.462+0000 7f9f96ffd640 10 client.4359.objecter ms_dispatch 0x7f9fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:46.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.462+0000 7f9f96ffd640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:46.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.462+0000 7f9f96ffd640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:46.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.462+0000 7f9f96ffd640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-25T19:43:46.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.462+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f9c050380 con 0x7f9fa80b4d60 2024-03-25T19:43:46.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.462+0000 7f9f96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:46.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.462+0000 7f9f96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:46.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.715+0000 7f9fac8f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.715+0000 7f9fac8f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.715+0000 7f9fac8f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:43:46.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.715+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fa80627b0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.738+0000 7f9f96ffd640 1 -- 172.21.15.18:0/4008687256 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f9f9c0522b0 con 0x7f9fa80b4d60 2024-03-25T19:43:46.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.738+0000 7f9f96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.738+0000 7f9f96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:46.739 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:43:46.739 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.740+0000 7f9fac8f4640 10 librados: watch_flush enter 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 10 librados: watch_flush exit 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 10 client.4359.objecter successfully canceled tick 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 >> v1:172.21.15.18:6800/1520000573 conn(0x7f9f8806df70 legacy=0x7f9f88070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 10 monclient: shutdown 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 >> v1:172.21.15.18:6789/0 conn(0x7f9fa80b4d60 legacy=0x7f9fa812f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 shutdown_connections 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 >> 172.21.15.18:0/4008687256 conn(0x7f9fa80617a0 msgr2=0x7f9fa813c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:46.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 shutdown_connections 2024-03-25T19:43:46.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 1 -- 172.21.15.18:0/4008687256 wait complete. 2024-03-25T19:43:46.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:46.741+0000 7f9fac8f4640 1 librados: shutdown 2024-03-25T19:43:46.754 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:47.214 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:14.744 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.273 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:14.803 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.308 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:14.838 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.346 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:14.842 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.348 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:14.879 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.355 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:14.852 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.361 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:14.892 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.391 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:14.922 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:47.396 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:14.892 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:53.560 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:53.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.674+0000 7f277959d640 10 monclient: get_monmap_and_config 2024-03-25T19:43:53.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.674+0000 7f277959d640 10 monclient: build_initial_monmap 2024-03-25T19:43:53.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.674+0000 7f277959d640 10 monclient: monmap: 2024-03-25T19:43:53.675 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:53.675 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:53.675 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:43:53.676543+0000 2024-03-25T19:43:53.675 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:43:53.676543+0000 2024-03-25T19:43:53.675 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:53.676 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:53.676 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:53.676 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:43:53.676 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:43:53.676 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.675+0000 7f277959d640 1 Processor -- start 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.675+0000 7f277959d640 1 -- start start 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.675+0000 7f277959d640 10 monclient: init 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 10 monclient(hunting): picked mon.noname-a con 0x7f27740a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 10 monclient(hunting): picked mon.noname-c con 0x7f277413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 10 monclient(hunting): picked mon.noname-b con 0x7f27741430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2774142520 con 0x7f27740a98f0 2024-03-25T19:43:53.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27740b28a0 con 0x7f277413f710 2024-03-25T19:43:53.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27740b2a30 con 0x7f27741430c0 2024-03-25T19:43:53.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:53.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.676+0000 7f277959d640 10 monclient(hunting): authenticate will time out at 5060.804075s 2024-03-25T19:43:53.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.685+0000 7f2772ffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f27740a98f0 0x7f27740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60408/0 (socket says 172.21.15.18:60408) 2024-03-25T19:43:53.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.685+0000 7f2772ffd640 1 -- 172.21.15.18:0/1086638497 learned_addr learned my addr 172.21.15.18:0/1086638497 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:53.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.707+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276788593 0 0) 0x7f2774142520 con 0x7f27740a98f0 2024-03-25T19:43:53.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.707+0000 7f27727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.707+0000 7f27727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.707+0000 7f27727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.707+0000 7f27727fc640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:43:53.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.707+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2760003520 con 0x7f27740a98f0 2024-03-25T19:43:53.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.710+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194824175 0 0) 0x7f27740b28a0 con 0x7f277413f710 2024-03-25T19:43:53.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.710+0000 7f27727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.710+0000 7f27727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.710+0000 7f27727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.710+0000 7f27727fc640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:43:53.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.710+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2774142520 con 0x7f277413f710 2024-03-25T19:43:53.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.716+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475511378 0 0) 0x7f27740b2a30 con 0x7f27741430c0 2024-03-25T19:43:53.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.716+0000 7f27727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.716+0000 7f27727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.716+0000 7f27727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.716+0000 7f27727fc640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:43:53.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.716+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27740b28a0 con 0x7f27741430c0 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3128686563 0 0) 0x7f2760003520 con 0x7f27740a98f0 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 >> v1:172.21.15.18:6790/0 conn(0x7f27741430c0 legacy=0x7f277414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 >> v1:172.21.15.53:6789/0 conn(0x7f277413f710 legacy=0x7f27741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 1 monclient: found mon.noname-a 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27740b44c0 con 0x7f27740a98f0 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2764001690 con 0x7f27740a98f0 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f277959d640 5 monclient: authenticate success, global_id 4362 2024-03-25T19:43:53.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f277959d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f27727fc640 10 monclient: dump: 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:53.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.730+0000 7f277959d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.758+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27640017f0 con 0x7f27740a98f0 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.758+0000 7f27727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.758+0000 7f27727fc640 1 -- 172.21.15.18:0/1086638497 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2764001c00 con 0x7f27740a98f0 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.758+0000 7f27727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.758+0000 7f27727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.758+0000 7f27727fc640 10 monclient: dump: 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:53.759 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:53.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.758+0000 7f277959d640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:53.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.758+0000 7f277959d640 4 set_mon_vals no callback set 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.759+0000 7f277959d640 10 monclient: shutdown 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.759+0000 7f277959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.759+0000 7f277959d640 1 -- 172.21.15.18:0/1086638497 >> v1:172.21.15.18:6789/0 conn(0x7f27740a98f0 legacy=0x7f27740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.759+0000 7f277959d640 1 -- 172.21.15.18:0/1086638497 shutdown_connections 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.759+0000 7f277959d640 1 -- 172.21.15.18:0/1086638497 >> 172.21.15.18:0/1086638497 conn(0x7f27740b7ea0 msgr2=0x7f27740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.759+0000 7f277959d640 1 -- 172.21.15.18:0/1086638497 shutdown_connections 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.759+0000 7f277959d640 1 -- 172.21.15.18:0/1086638497 wait complete. 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 10 monclient: build_initial_monmap 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 10 monclient: monmap: 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:53.761 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 1 librados: starting msgr at 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 1 librados: starting objecter 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 1 Processor -- start 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 1 -- start start 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 1 librados: setting wanted keys 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 1 librados: calling monclient init 2024-03-25T19:43:53.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.760+0000 7f277959d640 10 monclient: init 2024-03-25T19:43:53.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:53.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:53.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 10 monclient(hunting): picked mon.noname-c con 0x7f27740a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:43:53.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 10 monclient(hunting): picked mon.noname-b con 0x7f277413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:43:53.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 10 monclient(hunting): picked mon.noname-a con 0x7f27741430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:43:53.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2774138970 con 0x7f27741430c0 2024-03-25T19:43:53.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27741d7810 con 0x7f277413f710 2024-03-25T19:43:53.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2774139990 con 0x7f27740a98f0 2024-03-25T19:43:53.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:53.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.761+0000 7f277959d640 10 monclient(hunting): authenticate will time out at 5060.889252s 2024-03-25T19:43:53.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.768+0000 7f276bfff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f277413f710 0x7f27741d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:54850/0 (socket says 172.21.15.18:54850) 2024-03-25T19:43:53.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.768+0000 7f276bfff640 1 -- 172.21.15.18:0/3891975803 learned_addr learned my addr 172.21.15.18:0/3891975803 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:43:53.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.769+0000 7f276b7fe640 10 client.?.objecter ms_handle_connect 0x7f277413f710 2024-03-25T19:43:53.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.769+0000 7f276b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.775+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234850872 0 0) 0x7f27741d7810 con 0x7f277413f710 2024-03-25T19:43:53.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.775+0000 7f276b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.775+0000 7f276b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.775+0000 7f276b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.775+0000 7f276b7fe640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:43:53.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.775+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2754004eb0 con 0x7f277413f710 2024-03-25T19:43:53.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.780+0000 7f276b7fe640 10 client.?.objecter ms_handle_connect 0x7f27740a98f0 2024-03-25T19:43:53.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.780+0000 7f276b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.784+0000 7f276b7fe640 10 client.?.objecter ms_handle_connect 0x7f27741430c0 2024-03-25T19:43:53.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.784+0000 7f276b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.790+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2736938776 0 0) 0x7f2754004eb0 con 0x7f277413f710 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.790+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27741d7810 con 0x7f277413f710 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.790+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f275c001a30 con 0x7f277413f710 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.790+0000 7f276b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.790+0000 7f276b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.790+0000 7f276b7fe640 10 monclient(hunting): dump: 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.791 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.790+0000 7f276b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.791+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145841693 0 0) 0x7f2774139990 con 0x7f27740a98f0 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.791+0000 7f276b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.791+0000 7f276b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.791+0000 7f276b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.791+0000 7f276b7fe640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:43:53.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.791+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2754004eb0 con 0x7f27740a98f0 2024-03-25T19:43:53.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830716572 0 0) 0x7f27741d7810 con 0x7f277413f710 2024-03-25T19:43:53.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 >> v1:172.21.15.18:6790/0 conn(0x7f27740a98f0 legacy=0x7f2774138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 >> v1:172.21.15.18:6789/0 conn(0x7f27741430c0 legacy=0x7f27741d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:53.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 1 monclient: found mon.b 2024-03-25T19:43:53.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:53.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27741d8440 con 0x7f277413f710 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f276b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 5 monclient: authenticate success, global_id 4339 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 10 monclient: _renew_subs 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27741d41e0 con 0x7f277413f710 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 10 monclient: _renew_subs 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2774138d80 con 0x7f277413f710 2024-03-25T19:43:53.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.812+0000 7f277959d640 1 librados: init done 2024-03-25T19:43:53.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.813+0000 7f277959d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.814+0000 7f277959d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.814+0000 7f277959d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:53.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:53.814+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27741d79a0 con 0x7f277413f710 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.107+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f275c001cd0 con 0x7f277413f710 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.107+0000 7f276b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.107+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f275c003a60 con 0x7f277413f710 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.107+0000 7f276b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.107+0000 7f276b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.107+0000 7f276b7fe640 10 monclient: dump: 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:43:54.109 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:54.110 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:43:54.110 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:43:54.110 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:43:54.110 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:43:54.110 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:43:54.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.107+0000 7f2771ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:54.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.108+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f275c003c80 con 0x7f277413f710 2024-03-25T19:43:54.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.109+0000 7f2772ffd640 10 monclient: get_auth_request con 0x7f275406dd00 auth_method 0 2024-03-25T19:43:54.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.109+0000 7f276b7fe640 10 client.4339.objecter ms_handle_connect 0x7f275406dd00 2024-03-25T19:43:54.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.114+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f275c0548f0 con 0x7f277413f710 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.114+0000 7f276b7fe640 10 client.4339.objecter ms_dispatch 0x7f2774000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.114+0000 7f276b7fe640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.114+0000 7f276b7fe640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.114+0000 7f276b7fe640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.114+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f275c050400 con 0x7f277413f710 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.114+0000 7f276b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:54.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.114+0000 7f276b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:54.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.343+0000 7f277959d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:54.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.343+0000 7f277959d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:54.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.343+0000 7f277959d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:43:54.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.343+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27740627b0 con 0x7f277413f710 2024-03-25T19:43:54.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.382+0000 7f276b7fe640 1 -- 172.21.15.18:0/3891975803 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f275c052330 con 0x7f277413f710 2024-03-25T19:43:54.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.382+0000 7f276b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:54.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.382+0000 7f276b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:54.383 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:43:54.383 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 10 librados: watch_flush enter 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 10 librados: watch_flush exit 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 10 client.4339.objecter successfully canceled tick 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 >> v1:172.21.15.18:6800/1520000573 conn(0x7f275406dd00 legacy=0x7f27540701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 10 monclient: shutdown 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 >> v1:172.21.15.53:6789/0 conn(0x7f277413f710 legacy=0x7f27741d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 shutdown_connections 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 >> 172.21.15.18:0/3891975803 conn(0x7f27740b7ea0 msgr2=0x7f277413c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 shutdown_connections 2024-03-25T19:43:54.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 1 -- 172.21.15.18:0/3891975803 wait complete. 2024-03-25T19:43:54.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:43:54.384+0000 7f277959d640 1 librados: shutdown 2024-03-25T19:43:54.397 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:01.204 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:01.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.319+0000 7f4d047c6640 10 monclient: get_monmap_and_config 2024-03-25T19:44:01.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.319+0000 7f4d047c6640 10 monclient: build_initial_monmap 2024-03-25T19:44:01.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.319+0000 7f4d047c6640 10 monclient: monmap: 2024-03-25T19:44:01.320 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:01.320 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:01.320 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:44:01.321665+0000 2024-03-25T19:44:01.320 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:44:01.321665+0000 2024-03-25T19:44:01.321 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:01.321 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:01.321 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:01.321 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:44:01.321 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:44:01.321 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:01.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.320+0000 7f4d047c6640 1 Processor -- start 2024-03-25T19:44:01.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.320+0000 7f4d047c6640 1 -- start start 2024-03-25T19:44:01.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.320+0000 7f4d047c6640 10 monclient: init 2024-03-25T19:44:01.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:01.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:01.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:01.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:01.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:01.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc142520 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0b28a0 con 0x7f4cfc13f710 2024-03-25T19:44:01.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0b2a30 con 0x7f4cfc1430c0 2024-03-25T19:44:01.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:01.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.321+0000 7f4d047c6640 10 monclient(hunting): authenticate will time out at 5068.449362s 2024-03-25T19:44:01.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.337+0000 7f4d02d63640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f4cfc1430c0 0x7f4cfc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:50390/0 (socket says 172.21.15.18:50390) 2024-03-25T19:44:01.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.337+0000 7f4d02d63640 1 -- 172.21.15.18:0/1418294278 learned_addr learned my addr 172.21.15.18:0/1418294278 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.357+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820808144 0 0) 0x7f4cfc0b2a30 con 0x7f4cfc1430c0 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce8003520 con 0x7f4cfc1430c0 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24511859 0 0) 0x7f4cfc142520 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:44:01.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.358+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cfc0b2a30 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.362+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832506222 0 0) 0x7f4cfc0b28a0 con 0x7f4cfc13f710 2024-03-25T19:44:01.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.362+0000 7f4d01560640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.362+0000 7f4d01560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.362+0000 7f4d01560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.362+0000 7f4d01560640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:44:01.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.362+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cfc142520 con 0x7f4cfc13f710 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225435037 0 0) 0x7f4ce8003520 con 0x7f4cfc1430c0 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 >> v1:172.21.15.53:6789/0 conn(0x7f4cfc13f710 legacy=0x7f4cfc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 >> v1:172.21.15.18:6789/0 conn(0x7f4cfc0a98f0 legacy=0x7f4cfc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 10 monclient: _finish_hunting 0 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 1 monclient: found mon.noname-b 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0b44c0 con 0x7f4cfc1430c0 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 10 monclient: _finish_auth 0 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 10 monclient: _check_auth_tickets 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4cf4001a10 con 0x7f4cfc1430c0 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.369+0000 7f4d047c6640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.370+0000 7f4d047c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.370+0000 7f4d01560640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.370+0000 7f4d01560640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.370+0000 7f4d01560640 10 monclient: dump: 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:01.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.370+0000 7f4d047c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:01.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.380+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cf4001b70 con 0x7f4cfc1430c0 2024-03-25T19:44:01.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.380+0000 7f4d01560640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:01.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.380+0000 7f4d01560640 1 -- 172.21.15.18:0/1418294278 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4cf4001f80 con 0x7f4cfc1430c0 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.380+0000 7f4d01560640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.380+0000 7f4d01560640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.380+0000 7f4d01560640 10 monclient: dump: 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.380+0000 7f4d047c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:01.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.380+0000 7f4d047c6640 4 set_mon_vals no callback set 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.381+0000 7f4d047c6640 10 monclient: shutdown 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.381+0000 7f4d047c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.381+0000 7f4d047c6640 1 -- 172.21.15.18:0/1418294278 >> v1:172.21.15.18:6790/0 conn(0x7f4cfc1430c0 legacy=0x7f4cfc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.381+0000 7f4d047c6640 1 -- 172.21.15.18:0/1418294278 shutdown_connections 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.381+0000 7f4d047c6640 1 -- 172.21.15.18:0/1418294278 >> 172.21.15.18:0/1418294278 conn(0x7f4cfc0b7ea0 msgr2=0x7f4cfc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.381+0000 7f4d047c6640 1 -- 172.21.15.18:0/1418294278 shutdown_connections 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.381+0000 7f4d047c6640 1 -- 172.21.15.18:0/1418294278 wait complete. 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 10 monclient: build_initial_monmap 2024-03-25T19:44:01.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 10 monclient: monmap: 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 1 librados: starting msgr at 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 1 librados: starting objecter 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 1 Processor -- start 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 1 -- start start 2024-03-25T19:44:01.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 1 librados: setting wanted keys 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 1 librados: calling monclient init 2024-03-25T19:44:01.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.382+0000 7f4d047c6640 10 monclient: init 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc13f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc136ad0 con 0x7f4cfc1430c0 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0b7520 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc135810 con 0x7f4cfc13f710 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:01.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.383+0000 7f4d047c6640 10 monclient(hunting): authenticate will time out at 5068.511188s 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.392+0000 7f4d02562640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4cfc0a98f0 0x7f4cfc135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:50826/0 (socket says 172.21.15.18:50826) 2024-03-25T19:44:01.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.392+0000 7f4d02562640 1 -- 172.21.15.18:0/1685208694 learned_addr learned my addr 172.21.15.18:0/1685208694 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.393+0000 7f4cf2ffd640 10 client.?.objecter ms_handle_connect 0x7f4cfc0a98f0 2024-03-25T19:44:01.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.393+0000 7f4cf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.401+0000 7f4cf2ffd640 10 client.?.objecter ms_handle_connect 0x7f4cfc1430c0 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.401+0000 7f4cf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.406+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053490912 0 0) 0x7f4cfc0b7520 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.406+0000 7f4cf2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.406+0000 7f4cf2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.406+0000 7f4cf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.406+0000 7f4cf2ffd640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:44:01.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.406+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce0004eb0 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.414+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1289362184 0 0) 0x7f4ce0004eb0 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.414+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cfc0b7520 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.414+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4cf8001a30 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.414+0000 7f4cf2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.414+0000 7f4cf2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:01.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.414+0000 7f4cf2ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:01.415 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:01.415 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:01.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.414+0000 7f4cf2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:01.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.416+0000 7f4cf2ffd640 10 client.?.objecter ms_handle_connect 0x7f4cfc13f710 2024-03-25T19:44:01.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.416+0000 7f4cf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1951117130 0 0) 0x7f4cfc0b7520 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 >> v1:172.21.15.18:6790/0 conn(0x7f4cfc13f710 legacy=0x7f4cfc0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 >> v1:172.21.15.18:6789/0 conn(0x7f4cfc1430c0 legacy=0x7f4cfc130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 1 monclient: found mon.b 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0b6180 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4cf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:01.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:44:01.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 10 monclient: _renew_subs 2024-03-25T19:44:01.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:01.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cfc0b7a40 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:01.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 10 monclient: _renew_subs 2024-03-25T19:44:01.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:01.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cfc131840 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.428+0000 7f4d047c6640 1 librados: init done 2024-03-25T19:44:01.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.429+0000 7f4d047c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.429+0000 7f4d047c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.429+0000 7f4d047c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:01.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.429+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cfc136f60 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.754+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cf8001d10 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.754+0000 7f4cf2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:01.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.754+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f4cf80038c0 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.754+0000 7f4cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.754+0000 7f4cf2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.754+0000 7f4cf2ffd640 10 monclient: dump: 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:01.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.754+0000 7f4d00d5f640 4 set_mon_vals no callback set 2024-03-25T19:44:01.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.755+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f4cf8003ae0 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.756+0000 7f4d01d61640 10 monclient: get_auth_request con 0x7f4ce006dd00 auth_method 0 2024-03-25T19:44:01.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.756+0000 7f4cf2ffd640 10 client.4345.objecter ms_handle_connect 0x7f4ce006dd00 2024-03-25T19:44:01.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.760+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f4cf8054770 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.761+0000 7f4cf2ffd640 10 client.4345.objecter ms_dispatch 0x7f4cfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:01.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.761+0000 7f4cf2ffd640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:01.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.761+0000 7f4cf2ffd640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:01.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.761+0000 7f4cf2ffd640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-25T19:44:01.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.761+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4cf8050280 con 0x7f4cfc0a98f0 2024-03-25T19:44:01.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.761+0000 7f4cf2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:01.761+0000 7f4cf2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:02.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.030+0000 7f4d047c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:02.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.030+0000 7f4d047c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:02.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.030+0000 7f4d047c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:02.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.030+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cfc0b6310 con 0x7f4cfc0a98f0 2024-03-25T19:44:02.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.129+0000 7f4cf2ffd640 1 -- 172.21.15.18:0/1685208694 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f4cf80521b0 con 0x7f4cfc0a98f0 2024-03-25T19:44:02.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.129+0000 7f4cf2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:02.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.129+0000 7f4cf2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:02.129 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:44:02.129 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:02.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 10 librados: watch_flush enter 2024-03-25T19:44:02.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 10 librados: watch_flush exit 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 10 client.4345.objecter successfully canceled tick 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 >> v1:172.21.15.18:6800/1520000573 conn(0x7f4ce006dd00 legacy=0x7f4ce00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 10 monclient: shutdown 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 >> v1:172.21.15.53:6789/0 conn(0x7f4cfc0a98f0 legacy=0x7f4cfc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 shutdown_connections 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.131+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 >> 172.21.15.18:0/1685208694 conn(0x7f4cfc0b7ea0 msgr2=0x7f4cfc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.132+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 shutdown_connections 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.132+0000 7f4d047c6640 1 -- 172.21.15.18:0/1685208694 wait complete. 2024-03-25T19:44:02.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:02.132+0000 7f4d047c6640 1 librados: shutdown 2024-03-25T19:44:02.144 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:07.355 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:34.884 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:07.360 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:34.890 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:07.380 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:34.911 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:07.409 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:34.905 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:07.411 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:34.942 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:07.413 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:34.910 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:07.421 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:34.952 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:07.446 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:34.942 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:07.457 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:34.987 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:08.951 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:09.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.066+0000 7fea463f4640 10 monclient: get_monmap_and_config 2024-03-25T19:44:09.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.066+0000 7fea463f4640 10 monclient: build_initial_monmap 2024-03-25T19:44:09.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.066+0000 7fea463f4640 10 monclient: monmap: 2024-03-25T19:44:09.066 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:44:09.067742+0000 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:44:09.067742+0000 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:44:09.067 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:09.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.067+0000 7fea463f4640 1 Processor -- start 2024-03-25T19:44:09.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.067+0000 7fea463f4640 1 -- start start 2024-03-25T19:44:09.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.067+0000 7fea463f4640 10 monclient: init 2024-03-25T19:44:09.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:09.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:09.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 10 monclient(hunting): picked mon.noname-a con 0x7fea400a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:09.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 10 monclient(hunting): picked mon.noname-c con 0x7fea4013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:09.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 10 monclient(hunting): picked mon.noname-b con 0x7fea401430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:09.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea40142520 con 0x7fea400a98f0 2024-03-25T19:44:09.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea400b28a0 con 0x7fea4013f710 2024-03-25T19:44:09.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea400b2a30 con 0x7fea401430c0 2024-03-25T19:44:09.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:09.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.068+0000 7fea463f4640 10 monclient(hunting): authenticate will time out at 5076.195305s 2024-03-25T19:44:09.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.071+0000 7fea3ffff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fea400a98f0 0x7fea400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:49942/0 (socket says 172.21.15.18:49942) 2024-03-25T19:44:09.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.071+0000 7fea3ffff640 1 -- 172.21.15.18:0/554842258 learned_addr learned my addr 172.21.15.18:0/554842258 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:09.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.081+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651860403 0 0) 0x7fea40142520 con 0x7fea400a98f0 2024-03-25T19:44:09.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.081+0000 7fea3effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:09.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.081+0000 7fea3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:09.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.081+0000 7fea3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:09.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.081+0000 7fea3effd640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:44:09.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.081+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea28003520 con 0x7fea400a98f0 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 670789979 0 0) 0x7fea28003520 con 0x7fea400a98f0 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 >> v1:172.21.15.18:6790/0 conn(0x7fea401430c0 legacy=0x7fea4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 >> v1:172.21.15.53:6789/0 conn(0x7fea4013f710 legacy=0x7fea401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 1 monclient: found mon.noname-a 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea400b44c0 con 0x7fea400a98f0 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 10 monclient: _finish_auth 0 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea30001690 con 0x7fea400a98f0 2024-03-25T19:44:09.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea3effd640 10 monclient: dump: 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea463f4640 5 monclient: authenticate success, global_id 4374 2024-03-25T19:44:09.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.092+0000 7fea463f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:09.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea300017f0 con 0x7fea400a98f0 2024-03-25T19:44:09.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:09.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea3effd640 1 -- 172.21.15.18:0/554842258 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea30001c40 con 0x7fea400a98f0 2024-03-25T19:44:09.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:09.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea3effd640 10 monclient: dump: 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea463f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:09.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea463f4640 4 set_mon_vals no callback set 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea463f4640 10 monclient: shutdown 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea463f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea463f4640 1 -- 172.21.15.18:0/554842258 >> v1:172.21.15.18:6789/0 conn(0x7fea400a98f0 legacy=0x7fea400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea463f4640 1 -- 172.21.15.18:0/554842258 shutdown_connections 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.102+0000 7fea463f4640 1 -- 172.21.15.18:0/554842258 >> 172.21.15.18:0/554842258 conn(0x7fea400b7ea0 msgr2=0x7fea400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.103+0000 7fea463f4640 1 -- 172.21.15.18:0/554842258 shutdown_connections 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.103+0000 7fea463f4640 1 -- 172.21.15.18:0/554842258 wait complete. 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.103+0000 7fea463f4640 10 monclient: build_initial_monmap 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.103+0000 7fea463f4640 10 monclient: monmap: 2024-03-25T19:44:09.104 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.104+0000 7fea463f4640 1 librados: starting msgr at 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.104+0000 7fea463f4640 1 librados: starting objecter 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.104+0000 7fea463f4640 1 Processor -- start 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.104+0000 7fea463f4640 1 -- start start 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.104+0000 7fea463f4640 1 librados: setting wanted keys 2024-03-25T19:44:09.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.104+0000 7fea463f4640 1 librados: calling monclient init 2024-03-25T19:44:09.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.104+0000 7fea463f4640 10 monclient: init 2024-03-25T19:44:09.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 10 monclient(hunting): picked mon.noname-c con 0x7fea400a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 10 monclient(hunting): picked mon.noname-b con 0x7fea4013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 10 monclient(hunting): picked mon.noname-a con 0x7fea401430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea401323a0 con 0x7fea401430c0 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4014ddb0 con 0x7fea4013f710 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4014e090 con 0x7fea400a98f0 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:09.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.105+0000 7fea463f4640 10 monclient(hunting): authenticate will time out at 5076.232098s 2024-03-25T19:44:09.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.111+0000 7fea3f7fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fea4013f710 0x7fea40132f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:44546/0 (socket says 172.21.15.18:44546) 2024-03-25T19:44:09.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.111+0000 7fea3f7fe640 1 -- 172.21.15.18:0/3462385978 learned_addr learned my addr 172.21.15.18:0/3462385978 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:09.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.114+0000 7fea3cff9640 10 client.?.objecter ms_handle_connect 0x7fea4013f710 2024-03-25T19:44:09.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.114+0000 7fea3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:09.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.122+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311597903 0 0) 0x7fea4014ddb0 con 0x7fea4013f710 2024-03-25T19:44:09.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.122+0000 7fea3cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:09.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.122+0000 7fea3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:09.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.122+0000 7fea3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:09.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.122+0000 7fea3cff9640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:44:09.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.122+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c004eb0 con 0x7fea4013f710 2024-03-25T19:44:09.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.122+0000 7fea3cff9640 10 client.?.objecter ms_handle_connect 0x7fea400a98f0 2024-03-25T19:44:09.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.122+0000 7fea3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:09.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.123+0000 7fea3cff9640 10 client.?.objecter ms_handle_connect 0x7fea401430c0 2024-03-25T19:44:09.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.123+0000 7fea3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:09.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.131+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3563957502 0 0) 0x7fea2c004eb0 con 0x7fea4013f710 2024-03-25T19:44:09.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.132+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea4014ddb0 con 0x7fea4013f710 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.132+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea34001a30 con 0x7fea4013f710 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.132+0000 7fea3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.132+0000 7fea3cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.132+0000 7fea3cff9640 10 monclient(hunting): dump: 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:09.132 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:09.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.132+0000 7fea3cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:09.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.138+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464556064 0 0) 0x7fea4014e090 con 0x7fea400a98f0 2024-03-25T19:44:09.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.138+0000 7fea3cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:09.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.138+0000 7fea3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:09.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.138+0000 7fea3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:09.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.138+0000 7fea3cff9640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:44:09.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.138+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c004eb0 con 0x7fea400a98f0 2024-03-25T19:44:09.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.146+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063386446 0 0) 0x7fea401323a0 con 0x7fea401430c0 2024-03-25T19:44:09.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.146+0000 7fea3cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:09.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.146+0000 7fea3cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:09.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.146+0000 7fea3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:09.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.146+0000 7fea3cff9640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:44:09.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.146+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea4014e090 con 0x7fea401430c0 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2149743271 0 0) 0x7fea4014ddb0 con 0x7fea4013f710 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 >> v1:172.21.15.18:6790/0 conn(0x7fea400a98f0 legacy=0x7fea40132830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 >> v1:172.21.15.18:6789/0 conn(0x7fea401430c0 legacy=0x7fea401e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 1 monclient: found mon.b 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea40134c20 con 0x7fea4013f710 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:09.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 5 monclient: authenticate success, global_id 4348 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 10 monclient: _renew_subs 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4014bf20 con 0x7fea4013f710 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 10 monclient: _renew_subs 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4014e690 con 0x7fea4013f710 2024-03-25T19:44:09.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.158+0000 7fea463f4640 1 librados: init done 2024-03-25T19:44:09.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.159+0000 7fea463f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:09.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.159+0000 7fea463f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:09.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.159+0000 7fea463f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:09.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.159+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4014c2c0 con 0x7fea4013f710 2024-03-25T19:44:09.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.458+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea34001cb0 con 0x7fea4013f710 2024-03-25T19:44:09.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.458+0000 7fea3cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.458+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea34003a60 con 0x7fea4013f710 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.458+0000 7fea3cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.458+0000 7fea3cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.458+0000 7fea3cff9640 10 monclient: dump: 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:09.459 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:09.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.458+0000 7fea3e7fc640 4 set_mon_vals no callback set 2024-03-25T19:44:09.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.459+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fea34003c40 con 0x7fea4013f710 2024-03-25T19:44:09.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.459+0000 7fea3ffff640 10 monclient: get_auth_request con 0x7fea2c06de60 auth_method 0 2024-03-25T19:44:09.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.460+0000 7fea3cff9640 10 client.4348.objecter ms_handle_connect 0x7fea2c06de60 2024-03-25T19:44:09.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.464+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fea34054870 con 0x7fea4013f710 2024-03-25T19:44:09.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.464+0000 7fea3cff9640 10 client.4348.objecter ms_dispatch 0x7fea40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:09.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.464+0000 7fea3cff9640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:09.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.464+0000 7fea3cff9640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:09.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.464+0000 7fea3cff9640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T19:44:09.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.465+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea34050380 con 0x7fea4013f710 2024-03-25T19:44:09.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.465+0000 7fea3cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:09.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.465+0000 7fea3cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.716+0000 7fea463f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.716+0000 7fea463f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.716+0000 7fea463f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.716+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea40132550 con 0x7fea4013f710 2024-03-25T19:44:09.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.877+0000 7fea3cff9640 1 -- 172.21.15.18:0/3462385978 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fea40132550 con 0x7fea4013f710 2024-03-25T19:44:09.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.877+0000 7fea3cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.877+0000 7fea3cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:09.877 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:44:09.877 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:09.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 10 librados: watch_flush enter 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 10 librados: watch_flush exit 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 10 client.4348.objecter successfully canceled tick 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 >> v1:172.21.15.18:6800/1520000573 conn(0x7fea2c06de60 legacy=0x7fea2c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 10 monclient: shutdown 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 >> v1:172.21.15.53:6789/0 conn(0x7fea4013f710 legacy=0x7fea40132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 shutdown_connections 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 >> 172.21.15.18:0/3462385978 conn(0x7fea400b7ea0 msgr2=0x7fea40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 shutdown_connections 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 1 -- 172.21.15.18:0/3462385978 wait complete. 2024-03-25T19:44:09.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:09.879+0000 7fea463f4640 1 librados: shutdown 2024-03-25T19:44:09.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:16.697 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:16.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.813+0000 7fb21ca8d640 10 monclient: get_monmap_and_config 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.813+0000 7fb21ca8d640 10 monclient: build_initial_monmap 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.813+0000 7fb21ca8d640 10 monclient: monmap: 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:44:16.814452+0000 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:44:16.814452+0000 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.814+0000 7fb21ca8d640 1 Processor -- start 2024-03-25T19:44:16.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.814+0000 7fb21ca8d640 1 -- start start 2024-03-25T19:44:16.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.814+0000 7fb21ca8d640 10 monclient: init 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2180a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 10 monclient(hunting): picked mon.noname-a con 0x7fb21813f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2181430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218142520 con 0x7fb21813f710 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2180b28a0 con 0x7fb2180a98f0 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2180b2a30 con 0x7fb2181430c0 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:16.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.815+0000 7fb21ca8d640 10 monclient(hunting): authenticate will time out at 5083.942040s 2024-03-25T19:44:16.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.816+0000 7fb21659c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb2180a98f0 0x7fb2180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:52650/0 (socket says 172.21.15.18:52650) 2024-03-25T19:44:16.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.816+0000 7fb21659c640 1 -- 172.21.15.18:0/1836028490 learned_addr learned my addr 172.21.15.18:0/1836028490 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:16.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.818+0000 7fb21659c640 1 -- 172.21.15.18:0/1836028490 >> v1:172.21.15.53:6789/0 conn(0x7fb2180a98f0 legacy=0x7fb2180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T19:44:16.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.818+0000 7fb21659c640 1 -- 172.21.15.18:0/1836028490 >> v1:172.21.15.53:6789/0 conn(0x7fb2180a98f0 legacy=0x7fb2180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:44:16.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.818+0000 7fb21659c640 1 --1- 172.21.15.18:0/1836028490 >> v1:172.21.15.53:6789/0 conn(0x7fb2180a98f0 0x7fb2180a9cd0 :-1 s=OPENED pgs=21008 cs=1 l=1).handle_message read tag failed 2024-03-25T19:44:16.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.818+0000 7fb21659c640 1 --1- 172.21.15.18:0/1836028490 >> v1:172.21.15.53:6789/0 conn(0x7fb2180a98f0 0x7fb2180a9cd0 :-1 s=OPENED pgs=21008 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:44:16.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.818+0000 7fb21559a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.53:6789/0 2024-03-25T19:44:16.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.879+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681982668 0 0) 0x7fb218142520 con 0x7fb21813f710 2024-03-25T19:44:16.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.879+0000 7fb21559a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.879+0000 7fb21559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.879+0000 7fb21559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.879+0000 7fb21559a640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:44:16.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.879+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb200003520 con 0x7fb21813f710 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.921+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057515181 0 0) 0x7fb200003520 con 0x7fb21813f710 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.921+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 >> v1:172.21.15.18:6790/0 conn(0x7fb2181430c0 legacy=0x7fb21814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.921+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 >> v1:172.21.15.53:6789/0 conn(0x7fb2180a98f0 legacy=0x7fb2180a9cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 10 monclient: _finish_hunting 0 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 1 monclient: found mon.noname-a 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2180b44c0 con 0x7fb21813f710 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 10 monclient: _finish_auth 0 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 10 monclient: _check_auth_tickets 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:16.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21ca8d640 5 monclient: authenticate success, global_id 4380 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21ca8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb20c001690 con 0x7fb21813f710 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21559a640 10 monclient: dump: 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:16.923 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:16.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.922+0000 7fb21ca8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.943+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb20c0017f0 con 0x7fb21813f710 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.943+0000 7fb21559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.943+0000 7fb21559a640 1 -- 172.21.15.18:0/1836028490 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb20c001c40 con 0x7fb21813f710 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.943+0000 7fb21ca8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.943+0000 7fb21ca8d640 4 set_mon_vals no callback set 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.943+0000 7fb21559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.943+0000 7fb21559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.943+0000 7fb21559a640 10 monclient: dump: 2024-03-25T19:44:16.943 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:16.944 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:16.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.944+0000 7fb21ca8d640 10 monclient: shutdown 2024-03-25T19:44:16.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.944+0000 7fb21ca8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.944+0000 7fb21ca8d640 1 -- 172.21.15.18:0/1836028490 >> v1:172.21.15.18:6789/0 conn(0x7fb21813f710 legacy=0x7fb2181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.944+0000 7fb21ca8d640 1 -- 172.21.15.18:0/1836028490 shutdown_connections 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.944+0000 7fb21ca8d640 1 -- 172.21.15.18:0/1836028490 >> 172.21.15.18:0/1836028490 conn(0x7fb2180b7ea0 msgr2=0x7fb2180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.944+0000 7fb21ca8d640 1 -- 172.21.15.18:0/1836028490 shutdown_connections 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.944+0000 7fb21ca8d640 1 -- 172.21.15.18:0/1836028490 wait complete. 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.945+0000 7fb21ca8d640 10 monclient: build_initial_monmap 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.945+0000 7fb21ca8d640 10 monclient: monmap: 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:44:16.947 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 1 librados: starting msgr at 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 1 librados: starting objecter 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 1 Processor -- start 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 1 -- start start 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 1 librados: setting wanted keys 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 1 librados: calling monclient init 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 10 monclient: init 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2180a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 10 monclient(hunting): picked mon.noname-c con 0x7fb21813f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.946+0000 7fb21ca8d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2181430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.947+0000 7fb21ca8d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2181daa20 con 0x7fb2180a98f0 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.947+0000 7fb21ca8d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb218137ea0 con 0x7fb2181430c0 2024-03-25T19:44:16.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.947+0000 7fb21ca8d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2180aba90 con 0x7fb21813f710 2024-03-25T19:44:16.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.947+0000 7fb21ca8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:16.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.947+0000 7fb21ca8d640 10 monclient(hunting): authenticate will time out at 5084.073773s 2024-03-25T19:44:16.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.954+0000 7fb216d9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb2181430c0 0x7fb2181e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:52688/0 (socket says 172.21.15.18:52688) 2024-03-25T19:44:16.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.954+0000 7fb216d9d640 1 -- 172.21.15.18:0/455434924 learned_addr learned my addr 172.21.15.18:0/455434924 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:16.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.955+0000 7fb206ffd640 10 client.?.objecter ms_handle_connect 0x7fb2181430c0 2024-03-25T19:44:16.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.955+0000 7fb206ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.959+0000 7fb206ffd640 10 client.?.objecter ms_handle_connect 0x7fb21813f710 2024-03-25T19:44:16.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.959+0000 7fb206ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.964+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361835265 0 0) 0x7fb218137ea0 con 0x7fb2181430c0 2024-03-25T19:44:16.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.964+0000 7fb206ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.964+0000 7fb206ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.964+0000 7fb206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.964+0000 7fb206ffd640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:44:16.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.964+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f8004eb0 con 0x7fb2181430c0 2024-03-25T19:44:16.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.970+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059327141 0 0) 0x7fb2180aba90 con 0x7fb21813f710 2024-03-25T19:44:16.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.970+0000 7fb206ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.970+0000 7fb206ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.970+0000 7fb206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.970+0000 7fb206ffd640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:44:16.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.970+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb218137ea0 con 0x7fb21813f710 2024-03-25T19:44:16.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.975+0000 7fb206ffd640 10 client.?.objecter ms_handle_connect 0x7fb2180a98f0 2024-03-25T19:44:16.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.975+0000 7fb206ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.983+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509640604 0 0) 0x7fb2181daa20 con 0x7fb2180a98f0 2024-03-25T19:44:16.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.983+0000 7fb206ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.983+0000 7fb206ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.983+0000 7fb206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.983+0000 7fb206ffd640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:44:16.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.983+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2180aba90 con 0x7fb2180a98f0 2024-03-25T19:44:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142250866 0 0) 0x7fb218137ea0 con 0x7fb21813f710 2024-03-25T19:44:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2181daa20 con 0x7fb21813f710 2024-03-25T19:44:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41476538 0 0) 0x7fb1f8004eb0 con 0x7fb2181430c0 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb218137ea0 con 0x7fb2181430c0 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb20c001b40 con 0x7fb21813f710 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:16.991 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb210001bb0 con 0x7fb2181430c0 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:16.990+0000 7fb206ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:16.992 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:17.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3068164489 0 0) 0x7fb218137ea0 con 0x7fb2181430c0 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 >> v1:172.21.15.18:6790/0 conn(0x7fb21813f710 legacy=0x7fb2181daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 >> v1:172.21.15.18:6789/0 conn(0x7fb2180a98f0 legacy=0x7fb218137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 1 monclient: found mon.b 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2181db5c0 con 0x7fb2181430c0 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb21ca8d640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb21ca8d640 10 monclient: _renew_subs 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb21ca8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.000+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb218138310 con 0x7fb2181430c0 2024-03-25T19:44:17.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:17.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 10 monclient: _renew_subs 2024-03-25T19:44:17.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:17.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2181e3340 con 0x7fb2181430c0 2024-03-25T19:44:17.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 1 librados: init done 2024-03-25T19:44:17.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:17.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:17.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:17.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.001+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2180627b0 con 0x7fb2181430c0 2024-03-25T19:44:17.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.289+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb210001d10 con 0x7fb2181430c0 2024-03-25T19:44:17.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.289+0000 7fb206ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:17.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.289+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fb210003b10 con 0x7fb2181430c0 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.289+0000 7fb206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.289+0000 7fb206ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.289+0000 7fb206ffd640 10 monclient: dump: 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:17.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.289+0000 7fb214d99640 4 set_mon_vals no callback set 2024-03-25T19:44:17.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.290+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fb210003d30 con 0x7fb2181430c0 2024-03-25T19:44:17.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.291+0000 7fb21659c640 10 monclient: get_auth_request con 0x7fb1f806e000 auth_method 0 2024-03-25T19:44:17.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.291+0000 7fb206ffd640 10 client.4351.objecter ms_handle_connect 0x7fb1f806e000 2024-03-25T19:44:17.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.296+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fb2100549b0 con 0x7fb2181430c0 2024-03-25T19:44:17.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.296+0000 7fb206ffd640 10 client.4351.objecter ms_dispatch 0x7fb218000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:17.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.296+0000 7fb206ffd640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:17.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.296+0000 7fb206ffd640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:17.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.296+0000 7fb206ffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T19:44:17.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.297+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2100502f0 con 0x7fb2181430c0 2024-03-25T19:44:17.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.297+0000 7fb206ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:17.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.297+0000 7fb206ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:17.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.553+0000 7fb21ca8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:17.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.553+0000 7fb21ca8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:17.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.553+0000 7fb21ca8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:17.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.553+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2181dabd0 con 0x7fb2181430c0 2024-03-25T19:44:17.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.573+0000 7fb206ffd640 1 -- 172.21.15.18:0/455434924 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fb2181dabd0 con 0x7fb2181430c0 2024-03-25T19:44:17.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.573+0000 7fb206ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:17.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.573+0000 7fb206ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:17.573 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:44:17.573 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.575+0000 7fb21ca8d640 10 librados: watch_flush enter 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.575+0000 7fb21ca8d640 10 librados: watch_flush exit 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.575+0000 7fb21ca8d640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.575+0000 7fb21ca8d640 10 client.4351.objecter successfully canceled tick 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 >> v1:172.21.15.18:6800/1520000573 conn(0x7fb1f806e000 legacy=0x7fb1f80704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 10 monclient: shutdown 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 >> v1:172.21.15.53:6789/0 conn(0x7fb2181430c0 legacy=0x7fb2181e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 shutdown_connections 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 >> 172.21.15.18:0/455434924 conn(0x7fb2180b7ea0 msgr2=0x7fb21813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:17.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 shutdown_connections 2024-03-25T19:44:17.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 1 -- 172.21.15.18:0/455434924 wait complete. 2024-03-25T19:44:17.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:17.576+0000 7fb21ca8d640 1 librados: shutdown 2024-03-25T19:44:17.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:24.395 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:24.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.514+0000 7f0857c8d640 10 monclient: get_monmap_and_config 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.514+0000 7f0857c8d640 10 monclient: build_initial_monmap 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.514+0000 7f0857c8d640 10 monclient: monmap: 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:44:24.515028+0000 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:44:24.515028+0000 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:44:24.514 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.514+0000 7f0857c8d640 1 Processor -- start 2024-03-25T19:44:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 1 -- start start 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 10 monclient: init 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f08500b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f08500b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f085014be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500af940 con 0x7f08500b1730 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500b0a70 con 0x7f08500b4d60 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500b0d50 con 0x7f085014be90 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:24.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.515+0000 7f0857c8d640 10 monclient(hunting): authenticate will time out at 5091.642433s 2024-03-25T19:44:24.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.525+0000 7f0855228640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f08500b1730 0x7f08500b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:52470/0 (socket says 172.21.15.18:52470) 2024-03-25T19:44:24.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.525+0000 7f0855228640 1 -- 172.21.15.18:0/795542130 learned_addr learned my addr 172.21.15.18:0/795542130 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.537+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768214958 0 0) 0x7f08500b0a70 con 0x7f08500b4d60 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.537+0000 7f0854a27640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.537+0000 7f0854a27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.537+0000 7f0854a27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.537+0000 7f0854a27640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:44:24.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.537+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f083c003520 con 0x7f08500b4d60 2024-03-25T19:44:24.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.542+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149119742 0 0) 0x7f08500b0d50 con 0x7f085014be90 2024-03-25T19:44:24.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.542+0000 7f0854a27640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.542+0000 7f0854a27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.542+0000 7f0854a27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.542+0000 7f0854a27640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:44:24.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.542+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08500b0a70 con 0x7f085014be90 2024-03-25T19:44:24.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1538160691 0 0) 0x7f08500b0a70 con 0x7f085014be90 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 >> v1:172.21.15.53:6789/0 conn(0x7f08500b4d60 legacy=0x7f085013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 >> v1:172.21.15.18:6789/0 conn(0x7f08500b1730 legacy=0x7f08500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 10 monclient: _finish_hunting 0 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 1 monclient: found mon.noname-b 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500b1030 con 0x7f085014be90 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 10 monclient: _finish_auth 0 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 10 monclient: _check_auth_tickets 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0854a27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0857c8d640 5 monclient: authenticate success, global_id 4358 2024-03-25T19:44:24.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.547+0000 7f0857c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.548+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f08480019b0 con 0x7f085014be90 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.548+0000 7f0854a27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.548+0000 7f0854a27640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.548+0000 7f0854a27640 10 monclient: dump: 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.548 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:24.549 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:24.549 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:24.549 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:24.549 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.548+0000 7f0857c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.561+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0848001c40 con 0x7f085014be90 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.561+0000 7f0854a27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.561+0000 7f0854a27640 1 -- 172.21.15.18:0/795542130 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f0848001fd0 con 0x7f085014be90 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.561+0000 7f0857c8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.561+0000 7f0857c8d640 4 set_mon_vals no callback set 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.561+0000 7f0854a27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.561+0000 7f0854a27640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.561+0000 7f0854a27640 10 monclient: dump: 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.561 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.562 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.562 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:24.562 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:24.562 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:24.562 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:24.562 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:24.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.562+0000 7f0857c8d640 10 monclient: shutdown 2024-03-25T19:44:24.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.562+0000 7f0857c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:24.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.562+0000 7f0857c8d640 1 -- 172.21.15.18:0/795542130 >> v1:172.21.15.18:6790/0 conn(0x7f085014be90 legacy=0x7f085014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.562+0000 7f0857c8d640 1 -- 172.21.15.18:0/795542130 shutdown_connections 2024-03-25T19:44:24.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.562+0000 7f0857c8d640 1 -- 172.21.15.18:0/795542130 >> 172.21.15.18:0/795542130 conn(0x7f08500617a0 msgr2=0x7f0850061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.562+0000 7f0857c8d640 1 -- 172.21.15.18:0/795542130 shutdown_connections 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.562+0000 7f0857c8d640 1 -- 172.21.15.18:0/795542130 wait complete. 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.563+0000 7f0857c8d640 10 monclient: build_initial_monmap 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.563+0000 7f0857c8d640 10 monclient: monmap: 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:44:24.564 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:44:24.565 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.563+0000 7f0857c8d640 1 librados: starting msgr at 2024-03-25T19:44:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.563+0000 7f0857c8d640 1 librados: starting objecter 2024-03-25T19:44:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 1 Processor -- start 2024-03-25T19:44:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 1 -- start start 2024-03-25T19:44:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 1 librados: setting wanted keys 2024-03-25T19:44:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 1 librados: calling monclient init 2024-03-25T19:44:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 10 monclient: init 2024-03-25T19:44:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f08500b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f08500b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f085014be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08501403f0 con 0x7f08500b4d60 2024-03-25T19:44:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0850135bc0 con 0x7f085014be90 2024-03-25T19:44:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0850140680 con 0x7f08500b1730 2024-03-25T19:44:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.564+0000 7f0857c8d640 10 monclient(hunting): authenticate will time out at 5091.691707s 2024-03-25T19:44:24.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.577+0000 7f0855228640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f08500b4d60 0x7f0850132f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:52508/0 (socket says 172.21.15.18:52508) 2024-03-25T19:44:24.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.577+0000 7f0855228640 1 -- 172.21.15.18:0/4037051290 learned_addr learned my addr 172.21.15.18:0/4037051290 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:24.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.578+0000 7f08467fc640 10 client.?.objecter ms_handle_connect 0x7f08500b4d60 2024-03-25T19:44:24.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.579+0000 7f08467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.580+0000 7f08467fc640 10 client.?.objecter ms_handle_connect 0x7f085014be90 2024-03-25T19:44:24.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.580+0000 7f08467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.596+0000 7f08467fc640 10 client.?.objecter ms_handle_connect 0x7f08500b1730 2024-03-25T19:44:24.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.596+0000 7f08467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.605+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301714696 0 0) 0x7f08501403f0 con 0x7f08500b4d60 2024-03-25T19:44:24.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.605+0000 7f08467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.605+0000 7f08467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.605+0000 7f08467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.605+0000 7f08467fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:44:24.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.605+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0834004eb0 con 0x7f08500b4d60 2024-03-25T19:44:24.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.607+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583974364 0 0) 0x7f0850140680 con 0x7f08500b1730 2024-03-25T19:44:24.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.607+0000 7f08467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.607+0000 7f08467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.607+0000 7f08467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.607+0000 7f08467fc640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:44:24.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.607+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08501403f0 con 0x7f08500b1730 2024-03-25T19:44:24.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.614+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626906737 0 0) 0x7f0850135bc0 con 0x7f085014be90 2024-03-25T19:44:24.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.614+0000 7f08467fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.614+0000 7f08467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.614+0000 7f08467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.614+0000 7f08467fc640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:44:24.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.614+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0850140680 con 0x7f085014be90 2024-03-25T19:44:24.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3857885908 0 0) 0x7f0834004eb0 con 0x7f08500b4d60 2024-03-25T19:44:24.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0850135bc0 con 0x7f08500b4d60 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f08400018b0 con 0x7f08500b4d60 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 10 monclient(hunting): dump: 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:24.618 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 419348848 0 0) 0x7f08501403f0 con 0x7f08500b1730 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0834004eb0 con 0x7f08500b1730 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f084c001950 con 0x7f08500b1730 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.617+0000 7f08467fc640 10 monclient(hunting): dump: 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:24.619 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:24.620 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:24.620 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:24.620 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:24.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.638+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1375276387 0 0) 0x7f0850140680 con 0x7f085014be90 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.638+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08501403f0 con 0x7f085014be90 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.638+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f0848001f50 con 0x7f085014be90 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.638+0000 7f08467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.638+0000 7f08467fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.638+0000 7f08467fc640 10 monclient(hunting): dump: 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:24.638 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:24.639 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:24.639 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:24.639 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1770774417 0 0) 0x7f0834004eb0 con 0x7f08500b1730 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 >> v1:172.21.15.53:6789/0 conn(0x7f085014be90 legacy=0x7f08501d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 >> v1:172.21.15.18:6789/0 conn(0x7f08500b4d60 legacy=0x7f0850132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 1 monclient: found mon.c 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0850140810 con 0x7f08500b1730 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f08467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:24.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f0857c8d640 5 monclient: authenticate success, global_id 4361 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f0857c8d640 10 monclient: _renew_subs 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f0857c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.643+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0850133630 con 0x7f08500b1730 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.644+0000 7f0857c8d640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.644+0000 7f0857c8d640 10 monclient: _renew_subs 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.644+0000 7f0857c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.644+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0850143300 con 0x7f08500b1730 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.644+0000 7f0857c8d640 1 librados: init done 2024-03-25T19:44:24.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.645+0000 7f0857c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.645+0000 7f0857c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.645+0000 7f0857c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:24.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.645+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08500627b0 con 0x7f08500b1730 2024-03-25T19:44:24.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.991+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f084c001d60 con 0x7f08500b1730 2024-03-25T19:44:24.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.991+0000 7f08467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:24.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.991+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f084c0037b0 con 0x7f08500b1730 2024-03-25T19:44:24.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.991+0000 7f08467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.991+0000 7f08467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:24.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.991+0000 7f08467fc640 10 monclient: dump: 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:24.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.991+0000 7f0847fff640 4 set_mon_vals no callback set 2024-03-25T19:44:24.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.992+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f084c0197c0 con 0x7f08500b1730 2024-03-25T19:44:24.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.993+0000 7f0855228640 10 monclient: get_auth_request con 0x7f083406e160 auth_method 0 2024-03-25T19:44:24.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.993+0000 7f08467fc640 10 client.4361.objecter ms_handle_connect 0x7f083406e160 2024-03-25T19:44:24.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.998+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f084c053510 con 0x7f08500b1730 2024-03-25T19:44:24.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.998+0000 7f08467fc640 10 client.4361.objecter ms_dispatch 0x7f0850000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:24.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.998+0000 7f08467fc640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:24.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.998+0000 7f08467fc640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:24.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.998+0000 7f08467fc640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-25T19:44:24.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.998+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f084c04ee50 con 0x7f08500b1730 2024-03-25T19:44:24.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.998+0000 7f08467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:24.998+0000 7f08467fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:25.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.255+0000 7f0857c8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:25.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.255+0000 7f0857c8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:25.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.255+0000 7f0857c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:25.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.255+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08500b0330 con 0x7f08500b1730 2024-03-25T19:44:25.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.282+0000 7f08467fc640 1 -- 172.21.15.18:0/4037051290 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f084c050f50 con 0x7f08500b1730 2024-03-25T19:44:25.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.282+0000 7f08467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:25.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.282+0000 7f08467fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:25.282 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:44:25.282 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 10 librados: watch_flush enter 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 10 librados: watch_flush exit 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 10 client.4361.objecter successfully canceled tick 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 >> v1:172.21.15.18:6800/1520000573 conn(0x7f083406e160 legacy=0x7f0834070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 10 monclient: shutdown 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 >> v1:172.21.15.18:6790/0 conn(0x7f08500b1730 legacy=0x7f0850141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:25.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 shutdown_connections 2024-03-25T19:44:25.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 >> 172.21.15.18:0/4037051290 conn(0x7f08500617a0 msgr2=0x7f0850061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:25.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 shutdown_connections 2024-03-25T19:44:25.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 1 -- 172.21.15.18:0/4037051290 wait complete. 2024-03-25T19:44:25.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:25.284+0000 7f0857c8d640 1 librados: shutdown 2024-03-25T19:44:25.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:27.412 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:54.942 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:27.420 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:54.951 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:27.458 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:54.989 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:27.466 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:54.997 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:27.479 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:55.010 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:27.486 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:54.982 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:27.490 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:54.987 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:27.519 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:55.050 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:27.519 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:03:55.015 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:32.101 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:32.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.217+0000 7ff5c04fb640 10 monclient: get_monmap_and_config 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.217+0000 7ff5c04fb640 10 monclient: build_initial_monmap 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.217+0000 7ff5c04fb640 10 monclient: monmap: 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:44:32.217957+0000 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:44:32.217957+0000 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:44:32.217 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.217+0000 7ff5c04fb640 1 Processor -- start 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.217+0000 7ff5c04fb640 1 -- start start 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.217+0000 7ff5c04fb640 10 monclient: init 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff5b80b4f40 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b813fd90 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff5b80b4540 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80aa0c0 con 0x7ff5b813fd90 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80ab170 con 0x7ff5b80b4f40 2024-03-25T19:44:32.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80a9ba0 con 0x7ff5b80b4540 2024-03-25T19:44:32.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:32.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.218+0000 7ff5c04fb640 10 monclient(hunting): authenticate will time out at 5099.345538s 2024-03-25T19:44:32.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.224+0000 7ff5be297640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff5b80b4f40 0x7ff5b813f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:42008/0 (socket says 172.21.15.18:42008) 2024-03-25T19:44:32.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.224+0000 7ff5be297640 1 -- 172.21.15.18:0/2425848155 learned_addr learned my addr 172.21.15.18:0/2425848155 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:32.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.241+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772292262 0 0) 0x7ff5b80ab170 con 0x7ff5b80b4f40 2024-03-25T19:44:32.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.241+0000 7ff5bd295640 10 monclient(hunting): _negotiate 2024-03-25T19:44:32.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.241+0000 7ff5bd295640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:32.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.241+0000 7ff5bd295640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:32.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.241+0000 7ff5bd295640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:44:32.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.241+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a4003520 con 0x7ff5b80b4f40 2024-03-25T19:44:32.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.251+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520900016 0 0) 0x7ff5b80a9ba0 con 0x7ff5b80b4540 2024-03-25T19:44:32.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.251+0000 7ff5bd295640 10 monclient(hunting): _negotiate 2024-03-25T19:44:32.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.251+0000 7ff5bd295640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:32.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.251+0000 7ff5bd295640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:32.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.251+0000 7ff5bd295640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:44:32.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.251+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b80ab170 con 0x7ff5b80b4540 2024-03-25T19:44:32.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 394438713 0 0) 0x7ff5a4003520 con 0x7ff5b80b4f40 2024-03-25T19:44:32.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 >> v1:172.21.15.18:6790/0 conn(0x7ff5b80b4540 legacy=0x7ff5b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 >> v1:172.21.15.18:6789/0 conn(0x7ff5b813fd90 legacy=0x7ff5b81423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 10 monclient: _finish_hunting 0 2024-03-25T19:44:32.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 1 monclient: found mon.noname-c 2024-03-25T19:44:32.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b80a9680 con 0x7ff5b80b4f40 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 10 monclient: _finish_auth 0 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 10 monclient: _check_auth_tickets 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff5b4001690 con 0x7ff5b80b4f40 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5bd295640 10 monclient: dump: 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:32.254 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:32.255 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:32.255 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:32.255 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:32.255 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:32.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5c04fb640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:44:32.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.253+0000 7ff5c04fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:32.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.264+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5b40017f0 con 0x7ff5b80b4f40 2024-03-25T19:44:32.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.264+0000 7ff5bd295640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:32.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.264+0000 7ff5bd295640 1 -- 172.21.15.18:0/2425848155 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff5b4001c00 con 0x7ff5b80b4f40 2024-03-25T19:44:32.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.264+0000 7ff5bd295640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:32.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.264+0000 7ff5bd295640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:32.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.264+0000 7ff5bd295640 10 monclient: dump: 2024-03-25T19:44:32.264 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.264+0000 7ff5c04fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:32.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.264+0000 7ff5c04fb640 4 set_mon_vals no callback set 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.265+0000 7ff5c04fb640 10 monclient: shutdown 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.265+0000 7ff5c04fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.265+0000 7ff5c04fb640 1 -- 172.21.15.18:0/2425848155 >> v1:172.21.15.53:6789/0 conn(0x7ff5b80b4f40 legacy=0x7ff5b813f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.265+0000 7ff5c04fb640 1 -- 172.21.15.18:0/2425848155 shutdown_connections 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.265+0000 7ff5c04fb640 1 -- 172.21.15.18:0/2425848155 >> 172.21.15.18:0/2425848155 conn(0x7ff5b80617a0 msgr2=0x7ff5b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.265+0000 7ff5c04fb640 1 -- 172.21.15.18:0/2425848155 shutdown_connections 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.265+0000 7ff5c04fb640 1 -- 172.21.15.18:0/2425848155 wait complete. 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.266+0000 7ff5c04fb640 10 monclient: build_initial_monmap 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.266+0000 7ff5c04fb640 10 monclient: monmap: 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:44:32.268 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.266+0000 7ff5c04fb640 1 librados: starting msgr at 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.266+0000 7ff5c04fb640 1 librados: starting objecter 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.266+0000 7ff5c04fb640 1 Processor -- start 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 1 -- start start 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 1 librados: setting wanted keys 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 1 librados: calling monclient init 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 10 monclient: init 2024-03-25T19:44:32.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 10 monclient(hunting): picked mon.noname-b con 0x7ff5b80b4540 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b80b4f40 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 10 monclient(hunting): picked mon.noname-c con 0x7ff5b813fd90 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80ac360 con 0x7ff5b80b4f40 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80709c0 con 0x7ff5b80b4540 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b8138e80 con 0x7ff5b813fd90 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.267+0000 7ff5c04fb640 10 monclient(hunting): authenticate will time out at 5099.394610s 2024-03-25T19:44:32.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.270+0000 7ff5be297640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff5b80b4540 0x7ff5b8138770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:42042/0 (socket says 172.21.15.18:42042) 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.270+0000 7ff5be297640 1 -- 172.21.15.18:0/475388404 learned_addr learned my addr 172.21.15.18:0/475388404 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.272+0000 7ff5aeffd640 10 client.?.objecter ms_handle_connect 0x7ff5b813fd90 2024-03-25T19:44:32.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.272+0000 7ff5aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:32.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.275+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692589345 0 0) 0x7ff5b8138e80 con 0x7ff5b813fd90 2024-03-25T19:44:32.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.275+0000 7ff5aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:32.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.275+0000 7ff5aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:32.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.275+0000 7ff5aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:32.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.275+0000 7ff5aeffd640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:44:32.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.275+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c004eb0 con 0x7ff5b813fd90 2024-03-25T19:44:32.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.277+0000 7ff5aeffd640 10 client.?.objecter ms_handle_connect 0x7ff5b80b4540 2024-03-25T19:44:32.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.277+0000 7ff5aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:32.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.281+0000 7ff5aeffd640 10 client.?.objecter ms_handle_connect 0x7ff5b80b4f40 2024-03-25T19:44:32.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.281+0000 7ff5aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:32.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.283+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183905222 0 0) 0x7ff5b80709c0 con 0x7ff5b80b4540 2024-03-25T19:44:32.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.283+0000 7ff5aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:32.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.283+0000 7ff5aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:32.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.283+0000 7ff5aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:32.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.283+0000 7ff5aeffd640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:44:32.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.283+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b8138e80 con 0x7ff5b80b4540 2024-03-25T19:44:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.292+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3976609743 0 0) 0x7ff59c004eb0 con 0x7ff5b813fd90 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.292+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5b80709c0 con 0x7ff5b813fd90 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.292+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff5b0001cb0 con 0x7ff5b813fd90 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.292+0000 7ff5aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.292+0000 7ff5aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.292+0000 7ff5aeffd640 10 monclient(hunting): dump: 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:32.292 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:32.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.292+0000 7ff5aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.298+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3005322371 0 0) 0x7ff5b8138e80 con 0x7ff5b80b4540 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.298+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff59c004eb0 con 0x7ff5b80b4540 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.298+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff5b4001b40 con 0x7ff5b80b4540 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.298+0000 7ff5aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.298+0000 7ff5aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.298+0000 7ff5aeffd640 10 monclient(hunting): dump: 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:32.298 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:32.299 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:32.299 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:32.299 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:32.299 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.306+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162004656 0 0) 0x7ff5b80ac360 con 0x7ff5b80b4f40 2024-03-25T19:44:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.306+0000 7ff5aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.306+0000 7ff5aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.306+0000 7ff5aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.306+0000 7ff5aeffd640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:44:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.306+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b8138e80 con 0x7ff5b80b4f40 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 175684757 0 0) 0x7ff59c004eb0 con 0x7ff5b80b4540 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 >> v1:172.21.15.18:6790/0 conn(0x7ff5b813fd90 legacy=0x7ff5b81dc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 >> v1:172.21.15.18:6789/0 conn(0x7ff5b80b4f40 legacy=0x7ff5b81dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 1 monclient: found mon.b 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8139010 con 0x7ff5b80b4540 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 5 monclient: authenticate success, global_id 4363 2024-03-25T19:44:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 10 monclient: _renew_subs 2024-03-25T19:44:32.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:32.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b8070ee0 con 0x7ff5b80b4540 2024-03-25T19:44:32.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:32.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 10 monclient: _renew_subs 2024-03-25T19:44:32.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:32.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b80713c0 con 0x7ff5b80b4540 2024-03-25T19:44:32.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.317+0000 7ff5c04fb640 1 librados: init done 2024-03-25T19:44:32.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.318+0000 7ff5c04fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:32.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.318+0000 7ff5c04fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:32.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.318+0000 7ff5c04fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:32.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.318+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b8139280 con 0x7ff5b80b4540 2024-03-25T19:44:32.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.637+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5b4004a90 con 0x7ff5b80b4540 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.637+0000 7ff5aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.637+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff5b4004da0 con 0x7ff5b80b4540 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.637+0000 7ff5aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.637+0000 7ff5aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.637+0000 7ff5aeffd640 10 monclient: dump: 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:32.637 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:32.638 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:32.638 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:32.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.637+0000 7ff5bca94640 4 set_mon_vals no callback set 2024-03-25T19:44:32.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.638+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ff5b401ad70 con 0x7ff5b80b4540 2024-03-25T19:44:32.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.638+0000 7ff5bda96640 10 monclient: get_auth_request con 0x7ff59c06df30 auth_method 0 2024-03-25T19:44:32.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.639+0000 7ff5aeffd640 10 client.4363.objecter ms_handle_connect 0x7ff59c06df30 2024-03-25T19:44:32.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.643+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7ff5b40548e0 con 0x7ff5b80b4540 2024-03-25T19:44:32.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.643+0000 7ff5aeffd640 10 client.4363.objecter ms_dispatch 0x7ff5b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:32.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.643+0000 7ff5aeffd640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:32.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.643+0000 7ff5aeffd640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:32.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.643+0000 7ff5aeffd640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T19:44:32.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.643+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5b40503f0 con 0x7ff5b80b4540 2024-03-25T19:44:32.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.643+0000 7ff5aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:32.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.643+0000 7ff5aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:32.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.904+0000 7ff5c04fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.904+0000 7ff5c04fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.904+0000 7ff5c04fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:44:32.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.904+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5b8070b50 con 0x7ff5b80b4540 2024-03-25T19:44:32.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.937+0000 7ff5aeffd640 1 -- 172.21.15.18:0/475388404 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7ff5b4052320 con 0x7ff5b80b4540 2024-03-25T19:44:32.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.937+0000 7ff5aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.937+0000 7ff5aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:32.937 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:44:32.937 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:32.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 10 librados: watch_flush enter 2024-03-25T19:44:32.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 10 librados: watch_flush exit 2024-03-25T19:44:32.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 10 client.4363.objecter successfully canceled tick 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 >> v1:172.21.15.18:6800/1520000573 conn(0x7ff59c06df30 legacy=0x7ff59c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 10 monclient: shutdown 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 >> v1:172.21.15.53:6789/0 conn(0x7ff5b80b4540 legacy=0x7ff5b8138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 shutdown_connections 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.939+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 >> 172.21.15.18:0/475388404 conn(0x7ff5b80617a0 msgr2=0x7ff5b80b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.940+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 shutdown_connections 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.940+0000 7ff5c04fb640 1 -- 172.21.15.18:0/475388404 wait complete. 2024-03-25T19:44:32.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:32.940+0000 7ff5c04fb640 1 librados: shutdown 2024-03-25T19:44:32.951 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:39.759 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.876+0000 7f816b12b640 10 monclient: get_monmap_and_config 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.876+0000 7f816b12b640 10 monclient: build_initial_monmap 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.876+0000 7f816b12b640 10 monclient: monmap: 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:44:39.877277+0000 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:44:39.877277+0000 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:39.876 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:39.877 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:39.877 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:44:39.877 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:44:39.877 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.877+0000 7f816b12b640 1 Processor -- start 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.877+0000 7f816b12b640 1 -- start start 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.877+0000 7f816b12b640 10 monclient: init 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 10 monclient(hunting): picked mon.noname-b con 0x7f81640b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 10 monclient(hunting): picked mon.noname-c con 0x7f81640b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 10 monclient(hunting): picked mon.noname-a con 0x7f816414be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:39.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81640af940 con 0x7f816414be90 2024-03-25T19:44:39.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81640b0a70 con 0x7f81640b1730 2024-03-25T19:44:39.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81640b0d50 con 0x7f81640b4d60 2024-03-25T19:44:39.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:39.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.878+0000 7f816b12b640 10 monclient(hunting): authenticate will time out at 5107.005127s 2024-03-25T19:44:39.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.884+0000 7f8163fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f81640b1730 0x7f81640b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:34816/0 (socket says 172.21.15.18:34816) 2024-03-25T19:44:39.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.884+0000 7f8163fff640 1 -- 172.21.15.18:0/3900951387 learned_addr learned my addr 172.21.15.18:0/3900951387 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:39.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.899+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625060304 0 0) 0x7f81640b0d50 con 0x7f81640b4d60 2024-03-25T19:44:39.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.899+0000 7f81637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.899+0000 7f81637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.899+0000 7f81637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.899+0000 7f81637fe640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:44:39.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.899+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8154003520 con 0x7f81640b4d60 2024-03-25T19:44:39.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.904+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587259114 0 0) 0x7f81640b0a70 con 0x7f81640b1730 2024-03-25T19:44:39.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.904+0000 7f81637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.904+0000 7f81637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.904+0000 7f81637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.904+0000 7f81637fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:44:39.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.904+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81640b0d50 con 0x7f81640b1730 2024-03-25T19:44:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 832918373 0 0) 0x7f81640b0d50 con 0x7f81640b1730 2024-03-25T19:44:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 >> v1:172.21.15.18:6790/0 conn(0x7f81640b4d60 legacy=0x7f816413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 >> v1:172.21.15.18:6789/0 conn(0x7f816414be90 legacy=0x7f816414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 1 monclient: found mon.noname-c 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81640b1030 con 0x7f81640b1730 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8150001670 con 0x7f81640b1730 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f816b12b640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f816b12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f81637fe640 10 monclient: dump: 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:39.920 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:39.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.919+0000 7f816b12b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.932+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81500017d0 con 0x7f81640b1730 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.932+0000 7f81637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.932+0000 7f81637fe640 1 -- 172.21.15.18:0/3900951387 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8150001ba0 con 0x7f81640b1730 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.932+0000 7f816b12b640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.932+0000 7f816b12b640 4 set_mon_vals no callback set 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.932+0000 7f81637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.932+0000 7f81637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.932+0000 7f81637fe640 10 monclient: dump: 2024-03-25T19:44:39.932 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:39.933 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:39.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.933+0000 7f816b12b640 10 monclient: shutdown 2024-03-25T19:44:39.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.933+0000 7f816b12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:39.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.933+0000 7f816b12b640 1 -- 172.21.15.18:0/3900951387 >> v1:172.21.15.53:6789/0 conn(0x7f81640b1730 legacy=0x7f81640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.933+0000 7f816b12b640 1 -- 172.21.15.18:0/3900951387 shutdown_connections 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.933+0000 7f816b12b640 1 -- 172.21.15.18:0/3900951387 >> 172.21.15.18:0/3900951387 conn(0x7f81640617a0 msgr2=0x7f8164061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.933+0000 7f816b12b640 1 -- 172.21.15.18:0/3900951387 shutdown_connections 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.933+0000 7f816b12b640 1 -- 172.21.15.18:0/3900951387 wait complete. 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.934+0000 7f816b12b640 10 monclient: build_initial_monmap 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.934+0000 7f816b12b640 10 monclient: monmap: 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:44:39.935 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.934+0000 7f816b12b640 1 librados: starting msgr at 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.934+0000 7f816b12b640 1 librados: starting objecter 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.934+0000 7f816b12b640 1 Processor -- start 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 1 -- start start 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 1 librados: setting wanted keys 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 1 librados: calling monclient init 2024-03-25T19:44:39.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 10 monclient: init 2024-03-25T19:44:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 10 monclient(hunting): picked mon.noname-a con 0x7f81640b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 10 monclient(hunting): picked mon.noname-c con 0x7f81640b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 10 monclient(hunting): picked mon.noname-b con 0x7f816414be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81641409a0 con 0x7f81640b1730 2024-03-25T19:44:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.935+0000 7f816b12b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81641355e0 con 0x7f816414be90 2024-03-25T19:44:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.936+0000 7f816b12b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8164135770 con 0x7f81640b4d60 2024-03-25T19:44:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.936+0000 7f816b12b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.936+0000 7f816b12b640 10 monclient(hunting): authenticate will time out at 5107.062750s 2024-03-25T19:44:39.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.939+0000 7f81696c8640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f816414be90 0x7f81641db480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:34856/0 (socket says 172.21.15.18:34856) 2024-03-25T19:44:39.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.939+0000 7f81696c8640 1 -- 172.21.15.18:0/3617659286 learned_addr learned my addr 172.21.15.18:0/3617659286 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:39.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.940+0000 7f81617fa640 10 client.?.objecter ms_handle_connect 0x7f816414be90 2024-03-25T19:44:39.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.940+0000 7f81617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.943+0000 7f81617fa640 10 client.?.objecter ms_handle_connect 0x7f81640b1730 2024-03-25T19:44:39.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.943+0000 7f81617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.945+0000 7f81617fa640 10 client.?.objecter ms_handle_connect 0x7f81640b4d60 2024-03-25T19:44:39.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:39.945+0000 7f81617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:40.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.001+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727362192 0 0) 0x7f81641409a0 con 0x7f81640b1730 2024-03-25T19:44:40.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.001+0000 7f81617fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:40.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.001+0000 7f81617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:40.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.001+0000 7f81617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:40.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.001+0000 7f81617fa640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:44:40.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.001+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8148004eb0 con 0x7f81640b1730 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.026+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 289024402 0 0) 0x7f8148004eb0 con 0x7f81640b1730 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.026+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81641409a0 con 0x7f81640b1730 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.026+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8158001950 con 0x7f81640b1730 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.026+0000 7f81617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.026+0000 7f81617fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.026+0000 7f81617fa640 10 monclient(hunting): dump: 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:40.026 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:40.027 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:40.027 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.027 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:40.027 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:40.027 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:40.027 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:40.027 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:40.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.026+0000 7f81617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1871801543 0 0) 0x7f81641409a0 con 0x7f81640b1730 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 >> v1:172.21.15.18:6790/0 conn(0x7f81640b4d60 legacy=0x7f81641dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 >> v1:172.21.15.53:6789/0 conn(0x7f816414be90 legacy=0x7f81641db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 1 monclient: found mon.a 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164137740 con 0x7f81640b1730 2024-03-25T19:44:40.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f81617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 10 monclient: _renew_subs 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8164137990 con 0x7f81640b1730 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 10 monclient: _renew_subs 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8164144da0 con 0x7f81640b1730 2024-03-25T19:44:40.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.032+0000 7f816b12b640 1 librados: init done 2024-03-25T19:44:40.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.033+0000 7f816b12b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:40.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.033+0000 7f816b12b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:40.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.033+0000 7f816b12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:40.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.033+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81640627b0 con 0x7f81640b1730 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.307+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8158003610 con 0x7f81640b1730 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.307+0000 7f81617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.307+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8158003a20 con 0x7f81640b1730 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.307+0000 7f81617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.307+0000 7f81617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.307+0000 7f81617fa640 10 monclient: dump: 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:40.307 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:40.308 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:40.308 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:40.308 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:40.308 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:40.308 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:40.308 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:40.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.307+0000 7f8162ffd640 4 set_mon_vals no callback set 2024-03-25T19:44:40.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.308+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f8158003c40 con 0x7f81640b1730 2024-03-25T19:44:40.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.308+0000 7f8163fff640 10 monclient: get_auth_request con 0x7f814806dd00 auth_method 0 2024-03-25T19:44:40.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.308+0000 7f81617fa640 10 client.4401.objecter ms_handle_connect 0x7f814806dd00 2024-03-25T19:44:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.312+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f8158053510 con 0x7f81640b1730 2024-03-25T19:44:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.312+0000 7f81617fa640 10 client.4401.objecter ms_dispatch 0x7f8164000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.312+0000 7f81617fa640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.312+0000 7f81617fa640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.312+0000 7f81617fa640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T19:44:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.312+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f815804f020 con 0x7f81640b1730 2024-03-25T19:44:40.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.312+0000 7f81617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:40.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.312+0000 7f81617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:40.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.534+0000 7f816b12b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:40.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.534+0000 7f816b12b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:40.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.534+0000 7f816b12b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:40.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.534+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f816413cd30 con 0x7f81640b1730 2024-03-25T19:44:40.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.561+0000 7f81617fa640 1 -- 172.21.15.18:0/3617659286 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f8158050f50 con 0x7f81640b1730 2024-03-25T19:44:40.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.561+0000 7f81617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:40.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.561+0000 7f81617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:40.561 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:44:40.561 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:40.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.564+0000 7f816b12b640 10 librados: watch_flush enter 2024-03-25T19:44:40.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.564+0000 7f816b12b640 10 librados: watch_flush exit 2024-03-25T19:44:40.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.564+0000 7f816b12b640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T19:44:40.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.564+0000 7f816b12b640 10 client.4401.objecter successfully canceled tick 2024-03-25T19:44:40.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.564+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 >> v1:172.21.15.18:6800/1520000573 conn(0x7f814806dd00 legacy=0x7f81480701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.564+0000 7f816b12b640 10 monclient: shutdown 2024-03-25T19:44:40.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.564+0000 7f816b12b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:40.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.564+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 >> v1:172.21.15.18:6789/0 conn(0x7f81640b1730 legacy=0x7f8164134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:40.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.565+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 shutdown_connections 2024-03-25T19:44:40.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.565+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 >> 172.21.15.18:0/3617659286 conn(0x7f81640617a0 msgr2=0x7f81640b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:40.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.565+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 shutdown_connections 2024-03-25T19:44:40.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.565+0000 7f816b12b640 1 -- 172.21.15.18:0/3617659286 wait complete. 2024-03-25T19:44:40.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:40.565+0000 7f816b12b640 1 librados: shutdown 2024-03-25T19:44:40.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:47.383 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:47.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.478+0000 7f0600f23640 10 monclient: get_monmap_and_config 2024-03-25T19:44:47.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.478+0000 7f0600f23640 10 monclient: build_initial_monmap 2024-03-25T19:44:47.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.478+0000 7f0600f23640 10 monclient: monmap: 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:44:47.479540+0000 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:44:47.479540+0000 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:44:47.479 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.479+0000 7f0600f23640 1 Processor -- start 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.479+0000 7f0600f23640 1 -- start start 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.479+0000 7f0600f23640 10 monclient: init 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 10 monclient(hunting): picked mon.noname-b con 0x7f05fc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 10 monclient(hunting): picked mon.noname-c con 0x7f05fc13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 10 monclient(hunting): picked mon.noname-a con 0x7f05fc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc142520 con 0x7f05fc1430c0 2024-03-25T19:44:47.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0b28a0 con 0x7f05fc13f710 2024-03-25T19:44:47.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0b2a30 con 0x7f05fc0a98f0 2024-03-25T19:44:47.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:47.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.480+0000 7f0600f23640 10 monclient(hunting): authenticate will time out at 5114.607508s 2024-03-25T19:44:47.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.491+0000 7f05fad9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f05fc1430c0 0x7f05fc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41572/0 (socket says 172.21.15.18:41572) 2024-03-25T19:44:47.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.491+0000 7f05fad9d640 1 -- 172.21.15.18:0/2441328179 learned_addr learned my addr 172.21.15.18:0/2441328179 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:47.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.512+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431604822 0 0) 0x7f05fc142520 con 0x7f05fc1430c0 2024-03-25T19:44:47.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.512+0000 7f05f959a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.512+0000 7f05f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.512+0000 7f05f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.512+0000 7f05f959a640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:44:47.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.512+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e4003520 con 0x7f05fc1430c0 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977541473 0 0) 0x7f05e4003520 con 0x7f05fc1430c0 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 >> v1:172.21.15.18:6790/0 conn(0x7f05fc0a98f0 legacy=0x7f05fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 >> v1:172.21.15.53:6789/0 conn(0x7f05fc13f710 legacy=0x7f05fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 10 monclient: _finish_hunting 0 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 1 monclient: found mon.noname-a 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc0b44c0 con 0x7f05fc1430c0 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 10 monclient: _finish_auth 0 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 10 monclient: _check_auth_tickets 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:47.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f05f4001a10 con 0x7f05fc1430c0 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f05f959a640 10 monclient: dump: 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:47.529 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:47.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f0600f23640 5 monclient: authenticate success, global_id 4404 2024-03-25T19:44:47.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.528+0000 7f0600f23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:47.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.560+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f4001b70 con 0x7f05fc1430c0 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.560+0000 7f05f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.560+0000 7f05f959a640 1 -- 172.21.15.18:0/2441328179 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f05f4001f80 con 0x7f05fc1430c0 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.560+0000 7f0600f23640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.560+0000 7f0600f23640 4 set_mon_vals no callback set 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.560+0000 7f05f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.560+0000 7f05f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.560+0000 7f05f959a640 10 monclient: dump: 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.560 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.561 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.561 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:47.561 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:47.561 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:47.561 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:47.561 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:47.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.561+0000 7f0600f23640 10 monclient: shutdown 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.561+0000 7f0600f23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.561+0000 7f0600f23640 1 -- 172.21.15.18:0/2441328179 >> v1:172.21.15.18:6789/0 conn(0x7f05fc1430c0 legacy=0x7f05fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.561+0000 7f0600f23640 1 -- 172.21.15.18:0/2441328179 shutdown_connections 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.561+0000 7f0600f23640 1 -- 172.21.15.18:0/2441328179 >> 172.21.15.18:0/2441328179 conn(0x7f05fc0b7ea0 msgr2=0x7f05fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.561+0000 7f0600f23640 1 -- 172.21.15.18:0/2441328179 shutdown_connections 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.561+0000 7f0600f23640 1 -- 172.21.15.18:0/2441328179 wait complete. 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 10 monclient: build_initial_monmap 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 10 monclient: monmap: 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:44:47.564 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 1 librados: starting msgr at 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 1 librados: starting objecter 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 1 Processor -- start 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 1 -- start start 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 1 librados: setting wanted keys 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 1 librados: calling monclient init 2024-03-25T19:44:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.562+0000 7f0600f23640 10 monclient: init 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 10 monclient(hunting): picked mon.noname-b con 0x7f05fc0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 10 monclient(hunting): picked mon.noname-a con 0x7f05fc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 10 monclient(hunting): picked mon.noname-c con 0x7f05fc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc139a10 con 0x7f05fc13f710 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc1d7880 con 0x7f05fc0a98f0 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc137890 con 0x7f05fc1430c0 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.563+0000 7f0600f23640 10 monclient(hunting): authenticate will time out at 5114.690558s 2024-03-25T19:44:47.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.571+0000 7f05fa59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f05fc0a98f0 0x7f05fc1d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:52000/0 (socket says 172.21.15.18:52000) 2024-03-25T19:44:47.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.571+0000 7f05fa59c640 1 -- 172.21.15.18:0/3695938612 learned_addr learned my addr 172.21.15.18:0/3695938612 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:47.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.572+0000 7f05eaffd640 10 client.?.objecter ms_handle_connect 0x7f05fc0a98f0 2024-03-25T19:44:47.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.572+0000 7f05eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.580+0000 7f05eaffd640 10 client.?.objecter ms_handle_connect 0x7f05fc1430c0 2024-03-25T19:44:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.580+0000 7f05eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:47.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.584+0000 7f05eaffd640 10 client.?.objecter ms_handle_connect 0x7f05fc13f710 2024-03-25T19:44:47.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.584+0000 7f05eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:47.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889860495 0 0) 0x7f05fc137890 con 0x7f05fc1430c0 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05dc004eb0 con 0x7f05fc1430c0 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633789684 0 0) 0x7f05fc1d7880 con 0x7f05fc0a98f0 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:44:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.589+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05fc137890 con 0x7f05fc0a98f0 2024-03-25T19:44:47.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.603+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426733832 0 0) 0x7f05fc139a10 con 0x7f05fc13f710 2024-03-25T19:44:47.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.603+0000 7f05eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:47.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.603+0000 7f05eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:47.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.603+0000 7f05eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:47.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.603+0000 7f05eaffd640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:44:47.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.603+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05fc1d7880 con 0x7f05fc13f710 2024-03-25T19:44:47.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.607+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374305572 0 0) 0x7f05fc137890 con 0x7f05fc0a98f0 2024-03-25T19:44:47.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.607+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05fc139a10 con 0x7f05fc0a98f0 2024-03-25T19:44:47.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.607+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f05ec0019b0 con 0x7f05fc0a98f0 2024-03-25T19:44:47.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.607+0000 7f05eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.607+0000 7f05eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:47.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.607+0000 7f05eaffd640 10 monclient(hunting): dump: 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:47.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.607+0000 7f05eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.612+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 519625193 0 0) 0x7f05dc004eb0 con 0x7f05fc1430c0 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.612+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05fc137890 con 0x7f05fc1430c0 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.612+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f05f4001e80 con 0x7f05fc1430c0 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.612+0000 7f05eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.612+0000 7f05eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.612+0000 7f05eaffd640 10 monclient(hunting): dump: 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:47.613 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1272591828 0 0) 0x7f05fc137890 con 0x7f05fc1430c0 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 >> v1:172.21.15.53:6789/0 conn(0x7f05fc0a98f0 legacy=0x7f05fc1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 >> v1:172.21.15.18:6789/0 conn(0x7f05fc13f710 legacy=0x7f05fc138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 1 monclient: found mon.c 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc1d84b0 con 0x7f05fc1430c0 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:47.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f05eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f0600f23640 5 monclient: authenticate success, global_id 4376 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f0600f23640 10 monclient: _renew_subs 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f0600f23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.622+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc1d51f0 con 0x7f05fc1430c0 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.623+0000 7f0600f23640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.623+0000 7f0600f23640 10 monclient: _renew_subs 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.623+0000 7f0600f23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.623+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc138c60 con 0x7f05fc1430c0 2024-03-25T19:44:47.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.623+0000 7f0600f23640 1 librados: init done 2024-03-25T19:44:47.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.624+0000 7f0600f23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:47.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.624+0000 7f0600f23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:47.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.624+0000 7f0600f23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:47.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.624+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc1d7a10 con 0x7f05fc1430c0 2024-03-25T19:44:47.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.939+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f4002020 con 0x7f05fc1430c0 2024-03-25T19:44:47.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.939+0000 7f05eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:47.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.939+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f05f4004fb0 con 0x7f05fc1430c0 2024-03-25T19:44:47.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.939+0000 7f05eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.939+0000 7f05eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.939+0000 7f05eaffd640 10 monclient: dump: 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:47.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.939+0000 7f05f8d99640 4 set_mon_vals no callback set 2024-03-25T19:44:47.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.940+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f05f401af80 con 0x7f05fc1430c0 2024-03-25T19:44:47.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.941+0000 7f05fa59c640 10 monclient: get_auth_request con 0x7f05dc06df70 auth_method 0 2024-03-25T19:44:47.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.941+0000 7f05eaffd640 10 client.4376.objecter ms_handle_connect 0x7f05dc06df70 2024-03-25T19:44:47.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.947+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f05f4054ad0 con 0x7f05fc1430c0 2024-03-25T19:44:47.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.947+0000 7f05eaffd640 10 client.4376.objecter ms_dispatch 0x7f05fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:47.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.947+0000 7f05eaffd640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:47.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.947+0000 7f05eaffd640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:47.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.947+0000 7f05eaffd640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T19:44:47.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.947+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05f4050410 con 0x7f05fc1430c0 2024-03-25T19:44:47.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.947+0000 7f05eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:47.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:47.947+0000 7f05eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:48.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.204+0000 7f0600f23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:48.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.204+0000 7f0600f23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:48.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.204+0000 7f0600f23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:44:48.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.204+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05fc0627b0 con 0x7f05fc1430c0 2024-03-25T19:44:48.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.230+0000 7f05eaffd640 1 -- 172.21.15.18:0/3695938612 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f05f4052510 con 0x7f05fc1430c0 2024-03-25T19:44:48.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.230+0000 7f05eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:48.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.230+0000 7f05eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:48.230 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:44:48.230 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 10 librados: watch_flush enter 2024-03-25T19:44:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 10 librados: watch_flush exit 2024-03-25T19:44:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T19:44:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 10 client.4376.objecter successfully canceled tick 2024-03-25T19:44:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 >> v1:172.21.15.18:6800/1520000573 conn(0x7f05dc06df70 legacy=0x7f05dc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 10 monclient: shutdown 2024-03-25T19:44:48.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:48.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 >> v1:172.21.15.18:6790/0 conn(0x7f05fc1430c0 legacy=0x7f05fc1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:48.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 shutdown_connections 2024-03-25T19:44:48.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 >> 172.21.15.18:0/3695938612 conn(0x7f05fc0b7ea0 msgr2=0x7f05fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:48.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 shutdown_connections 2024-03-25T19:44:48.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 1 -- 172.21.15.18:0/3695938612 wait complete. 2024-03-25T19:44:48.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:48.233+0000 7f0600f23640 1 librados: shutdown 2024-03-25T19:44:48.245 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:55.052 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:55.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.175+0000 7f8b9e655640 10 monclient: get_monmap_and_config 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.175+0000 7f8b9e655640 10 monclient: build_initial_monmap 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.175+0000 7f8b9e655640 10 monclient: monmap: 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:44:55.176613+0000 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:44:55.176613+0000 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:44:55.176 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.176+0000 7f8b9e655640 1 Processor -- start 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.176+0000 7f8b9e655640 1 -- start start 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.176+0000 7f8b9e655640 10 monclient: init 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 10 monclient(hunting): picked mon.noname-a con 0x7f8b980a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 10 monclient(hunting): picked mon.noname-b con 0x7f8b9813f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 10 monclient(hunting): picked mon.noname-c con 0x7f8b981430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b98142520 con 0x7f8b980a98f0 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980b28a0 con 0x7f8b981430c0 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980b2a30 con 0x7f8b9813f710 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:55.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.177+0000 7f8b9e655640 10 monclient(hunting): authenticate will time out at 5122.304440s 2024-03-25T19:44:55.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.186+0000 7f8b977fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f8b9813f710 0x7f8b981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57032/0 (socket says 172.21.15.18:57032) 2024-03-25T19:44:55.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.187+0000 7f8b977fe640 1 -- 172.21.15.18:0/1403353180 learned_addr learned my addr 172.21.15.18:0/1403353180 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:55.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.201+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216781738 0 0) 0x7f8b980b2a30 con 0x7f8b9813f710 2024-03-25T19:44:55.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.201+0000 7f8b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:55.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.201+0000 7f8b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:55.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.201+0000 7f8b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:55.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.201+0000 7f8b96ffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:44:55.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.201+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b80003520 con 0x7f8b9813f710 2024-03-25T19:44:55.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.213+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607262926 0 0) 0x7f8b980b28a0 con 0x7f8b981430c0 2024-03-25T19:44:55.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.213+0000 7f8b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:55.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.213+0000 7f8b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:55.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.213+0000 7f8b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:55.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.213+0000 7f8b96ffd640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:44:55.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.213+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b980b2a30 con 0x7f8b981430c0 2024-03-25T19:44:55.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.214+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933525444 0 0) 0x7f8b98142520 con 0x7f8b980a98f0 2024-03-25T19:44:55.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.214+0000 7f8b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:55.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.214+0000 7f8b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:55.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.214+0000 7f8b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:55.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.214+0000 7f8b96ffd640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:44:55.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.214+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b980b28a0 con 0x7f8b980a98f0 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.218+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4187800665 0 0) 0x7f8b80003520 con 0x7f8b9813f710 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 >> v1:172.21.15.53:6789/0 conn(0x7f8b981430c0 legacy=0x7f8b9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 >> v1:172.21.15.18:6789/0 conn(0x7f8b980a98f0 legacy=0x7f8b980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 1 monclient: found mon.noname-b 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980b44c0 con 0x7f8b9813f710 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8b8c001690 con 0x7f8b9813f710 2024-03-25T19:44:55.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b96ffd640 10 monclient: dump: 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b9e655640 5 monclient: authenticate success, global_id 4379 2024-03-25T19:44:55.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.219+0000 7f8b9e655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.241+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b8c0017f0 con 0x7f8b9813f710 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.241+0000 7f8b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.241+0000 7f8b96ffd640 1 -- 172.21.15.18:0/1403353180 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8b8c001c00 con 0x7f8b9813f710 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.241+0000 7f8b9e655640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.241+0000 7f8b9e655640 4 set_mon_vals no callback set 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.241+0000 7f8b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.241+0000 7f8b96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.241+0000 7f8b96ffd640 10 monclient: dump: 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:55.241 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:55.242 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:55.242 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:55.242 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:55.242 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:55.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.242+0000 7f8b9e655640 10 monclient: shutdown 2024-03-25T19:44:55.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.242+0000 7f8b9e655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:55.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.242+0000 7f8b9e655640 1 -- 172.21.15.18:0/1403353180 >> v1:172.21.15.18:6790/0 conn(0x7f8b9813f710 legacy=0x7f8b981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.242+0000 7f8b9e655640 1 -- 172.21.15.18:0/1403353180 shutdown_connections 2024-03-25T19:44:55.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.242+0000 7f8b9e655640 1 -- 172.21.15.18:0/1403353180 >> 172.21.15.18:0/1403353180 conn(0x7f8b980b7ea0 msgr2=0x7f8b980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:55.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.242+0000 7f8b9e655640 1 -- 172.21.15.18:0/1403353180 shutdown_connections 2024-03-25T19:44:55.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.242+0000 7f8b9e655640 1 -- 172.21.15.18:0/1403353180 wait complete. 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.243+0000 7f8b9e655640 10 monclient: build_initial_monmap 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.243+0000 7f8b9e655640 10 monclient: monmap: 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.244+0000 7f8b9e655640 1 librados: starting msgr at 2024-03-25T19:44:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.244+0000 7f8b9e655640 1 librados: starting objecter 2024-03-25T19:44:55.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.244+0000 7f8b9e655640 1 Processor -- start 2024-03-25T19:44:55.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.244+0000 7f8b9e655640 1 -- start start 2024-03-25T19:44:55.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.244+0000 7f8b9e655640 1 librados: setting wanted keys 2024-03-25T19:44:55.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.244+0000 7f8b9e655640 1 librados: calling monclient init 2024-03-25T19:44:55.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.244+0000 7f8b9e655640 10 monclient: init 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 10 monclient(hunting): picked mon.noname-c con 0x7f8b980a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 10 monclient(hunting): picked mon.noname-b con 0x7f8b9813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 10 monclient(hunting): picked mon.noname-a con 0x7f8b981430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980aba00 con 0x7f8b981430c0 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b981d7490 con 0x7f8b9813f710 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b981d8360 con 0x7f8b980a98f0 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:55.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.245+0000 7f8b9e655640 10 monclient(hunting): authenticate will time out at 5122.372100s 2024-03-25T19:44:55.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.264+0000 7f8b97fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f8b980a98f0 0x7f8b981386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57082/0 (socket says 172.21.15.18:57082) 2024-03-25T19:44:55.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.264+0000 7f8b97fff640 1 -- 172.21.15.18:0/2611584538 learned_addr learned my addr 172.21.15.18:0/2611584538 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:44:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.267+0000 7f8b94ff9640 10 client.?.objecter ms_handle_connect 0x7f8b980a98f0 2024-03-25T19:44:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.267+0000 7f8b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:55.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.269+0000 7f8b94ff9640 10 client.?.objecter ms_handle_connect 0x7f8b981430c0 2024-03-25T19:44:55.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.269+0000 7f8b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:55.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.275+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95183026 0 0) 0x7f8b981d8360 con 0x7f8b980a98f0 2024-03-25T19:44:55.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.275+0000 7f8b94ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:55.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.275+0000 7f8b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:55.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.275+0000 7f8b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:55.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.275+0000 7f8b94ff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:44:55.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.275+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b84004eb0 con 0x7f8b980a98f0 2024-03-25T19:44:55.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.276+0000 7f8b94ff9640 10 client.?.objecter ms_handle_connect 0x7f8b9813f710 2024-03-25T19:44:55.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.276+0000 7f8b94ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:55.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.280+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141759454 0 0) 0x7f8b980aba00 con 0x7f8b981430c0 2024-03-25T19:44:55.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.280+0000 7f8b94ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:55.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.280+0000 7f8b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:55.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.280+0000 7f8b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:55.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.280+0000 7f8b94ff9640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:44:55.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.280+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b981d8360 con 0x7f8b981430c0 2024-03-25T19:44:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.292+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988613244 0 0) 0x7f8b981d7490 con 0x7f8b9813f710 2024-03-25T19:44:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.292+0000 7f8b94ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:55.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.292+0000 7f8b94ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:55.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.292+0000 7f8b94ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:55.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.292+0000 7f8b94ff9640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:44:55.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.292+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b980aba00 con 0x7f8b9813f710 2024-03-25T19:44:55.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.297+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1512847088 0 0) 0x7f8b981d8360 con 0x7f8b981430c0 2024-03-25T19:44:55.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.297+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b981d7490 con 0x7f8b981430c0 2024-03-25T19:44:55.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.297+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8b90001b50 con 0x7f8b981430c0 2024-03-25T19:44:55.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.297+0000 7f8b94ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:55.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.297+0000 7f8b94ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.297+0000 7f8b94ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:55.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.297+0000 7f8b94ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:55.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3400882753 0 0) 0x7f8b981d7490 con 0x7f8b981430c0 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 >> v1:172.21.15.18:6790/0 conn(0x7f8b980a98f0 legacy=0x7f8b981386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 >> v1:172.21.15.53:6789/0 conn(0x7f8b9813f710 legacy=0x7f8b981d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 1 monclient: found mon.a 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980abc30 con 0x7f8b981430c0 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 5 monclient: authenticate success, global_id 4413 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 10 monclient: _renew_subs 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:55.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b98137940 con 0x7f8b981430c0 2024-03-25T19:44:55.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:55.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 10 monclient: _renew_subs 2024-03-25T19:44:55.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:55.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b98135220 con 0x7f8b981430c0 2024-03-25T19:44:55.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.306+0000 7f8b9e655640 1 librados: init done 2024-03-25T19:44:55.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.307+0000 7f8b9e655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:55.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.307+0000 7f8b9e655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:55.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.307+0000 7f8b9e655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:55.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.307+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b980627b0 con 0x7f8b981430c0 2024-03-25T19:44:55.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.620+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b90002080 con 0x7f8b981430c0 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.620+0000 7f8b94ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.620+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8b90003b10 con 0x7f8b981430c0 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.620+0000 7f8b94ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.620+0000 7f8b94ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.620+0000 7f8b94ff9640 10 monclient: dump: 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:55.620 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:44:55.621 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:44:55.621 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:44:55.621 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:44:55.621 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:44:55.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.620+0000 7f8b967fc640 4 set_mon_vals no callback set 2024-03-25T19:44:55.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.621+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f8b90019ae0 con 0x7f8b981430c0 2024-03-25T19:44:55.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.621+0000 7f8b97fff640 10 monclient: get_auth_request con 0x7f8b8406de20 auth_method 0 2024-03-25T19:44:55.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.621+0000 7f8b94ff9640 10 client.4413.objecter ms_handle_connect 0x7f8b8406de20 2024-03-25T19:44:55.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.626+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f8b900537c0 con 0x7f8b981430c0 2024-03-25T19:44:55.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.626+0000 7f8b94ff9640 10 client.4413.objecter ms_dispatch 0x7f8b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:55.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.626+0000 7f8b94ff9640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:55.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.626+0000 7f8b94ff9640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:55.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.626+0000 7f8b94ff9640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-25T19:44:55.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.626+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b9004f100 con 0x7f8b981430c0 2024-03-25T19:44:55.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.626+0000 7f8b94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:55.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.626+0000 7f8b94ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:55.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.874+0000 7f8b9e655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.874+0000 7f8b9e655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.874+0000 7f8b9e655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:44:55.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.874+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b981d7830 con 0x7f8b981430c0 2024-03-25T19:44:55.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.961+0000 7f8b94ff9640 1 -- 172.21.15.18:0/2611584538 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f8b90051200 con 0x7f8b981430c0 2024-03-25T19:44:55.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.961+0000 7f8b94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.961+0000 7f8b94ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:55.962 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:44:55.962 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 10 librados: watch_flush enter 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 10 librados: watch_flush exit 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 10 client.4413.objecter successfully canceled tick 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 >> v1:172.21.15.18:6800/1520000573 conn(0x7f8b8406de20 legacy=0x7f8b840702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 10 monclient: shutdown 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 >> v1:172.21.15.18:6789/0 conn(0x7f8b981430c0 legacy=0x7f8b98133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 shutdown_connections 2024-03-25T19:44:55.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 >> 172.21.15.18:0/2611584538 conn(0x7f8b980b7ea0 msgr2=0x7f8b98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:55.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 shutdown_connections 2024-03-25T19:44:55.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 1 -- 172.21.15.18:0/2611584538 wait complete. 2024-03-25T19:44:55.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:44:55.964+0000 7f8b9e655640 1 librados: shutdown 2024-03-25T19:44:55.976 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:02.783 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:02.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.895+0000 7fee5496b640 10 monclient: get_monmap_and_config 2024-03-25T19:45:02.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.895+0000 7fee5496b640 10 monclient: build_initial_monmap 2024-03-25T19:45:02.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.895+0000 7fee5496b640 10 monclient: monmap: 2024-03-25T19:45:02.895 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:02.895 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:02.895 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:45:02.896163+0000 2024-03-25T19:45:02.895 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:45:02.896163+0000 2024-03-25T19:45:02.895 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:02.896 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:02.896 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:02.896 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:45:02.896 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:45:02.896 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.896+0000 7fee5496b640 1 Processor -- start 2024-03-25T19:45:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.896+0000 7fee5496b640 1 -- start start 2024-03-25T19:45:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.896+0000 7fee5496b640 10 monclient: init 2024-03-25T19:45:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.896+0000 7fee5496b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.896+0000 7fee5496b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.896+0000 7fee5496b640 10 monclient(hunting): picked mon.noname-c con 0x7fee500a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.896+0000 7fee5496b640 10 monclient(hunting): picked mon.noname-b con 0x7fee5013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.896+0000 7fee5496b640 10 monclient(hunting): picked mon.noname-a con 0x7fee501430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:02.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.897+0000 7fee5496b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee50142520 con 0x7fee501430c0 2024-03-25T19:45:02.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.897+0000 7fee5496b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500b28a0 con 0x7fee500a98f0 2024-03-25T19:45:02.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.897+0000 7fee5496b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500b2a30 con 0x7fee5013f710 2024-03-25T19:45:02.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.897+0000 7fee5496b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:02.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.897+0000 7fee5496b640 10 monclient(hunting): authenticate will time out at 5130.023783s 2024-03-25T19:45:02.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.908+0000 7fee4ed9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fee501430c0 0x7fee5014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:43780/0 (socket says 172.21.15.18:43780) 2024-03-25T19:45:02.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.908+0000 7fee4ed9d640 1 -- 172.21.15.18:0/2051855267 learned_addr learned my addr 172.21.15.18:0/2051855267 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:02.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.929+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923287823 0 0) 0x7fee50142520 con 0x7fee501430c0 2024-03-25T19:45:02.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.929+0000 7fee4dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.929+0000 7fee4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.929+0000 7fee4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.929+0000 7fee4dd9b640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:45:02.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.929+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3c003520 con 0x7fee501430c0 2024-03-25T19:45:02.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1300721689 0 0) 0x7fee3c003520 con 0x7fee501430c0 2024-03-25T19:45:02.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 >> v1:172.21.15.18:6790/0 conn(0x7fee5013f710 legacy=0x7fee501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 >> v1:172.21.15.53:6789/0 conn(0x7fee500a98f0 legacy=0x7fee500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee500b44c0 con 0x7fee501430c0 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fee48001a10 con 0x7fee501430c0 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee5496b640 5 monclient: authenticate success, global_id 4416 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:03.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee4dd9b640 10 monclient: dump: 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:03.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:02.999+0000 7fee5496b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:03.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.009+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee48001b70 con 0x7fee501430c0 2024-03-25T19:45:03.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.009+0000 7fee4dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:03.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.009+0000 7fee4dd9b640 1 -- 172.21.15.18:0/2051855267 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fee48001f80 con 0x7fee501430c0 2024-03-25T19:45:03.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.009+0000 7fee5496b640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.009+0000 7fee5496b640 4 set_mon_vals no callback set 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.009+0000 7fee4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.009+0000 7fee4dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.009+0000 7fee4dd9b640 10 monclient: dump: 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:03.010 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:03.011 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:03.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.010+0000 7fee5496b640 10 monclient: shutdown 2024-03-25T19:45:03.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.010+0000 7fee5496b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:03.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.010+0000 7fee5496b640 1 -- 172.21.15.18:0/2051855267 >> v1:172.21.15.18:6789/0 conn(0x7fee501430c0 legacy=0x7fee5014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.010+0000 7fee5496b640 1 -- 172.21.15.18:0/2051855267 shutdown_connections 2024-03-25T19:45:03.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.010+0000 7fee5496b640 1 -- 172.21.15.18:0/2051855267 >> 172.21.15.18:0/2051855267 conn(0x7fee500b7ea0 msgr2=0x7fee500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:03.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.010+0000 7fee5496b640 1 -- 172.21.15.18:0/2051855267 shutdown_connections 2024-03-25T19:45:03.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.010+0000 7fee5496b640 1 -- 172.21.15.18:0/2051855267 wait complete. 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.011+0000 7fee5496b640 10 monclient: build_initial_monmap 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.011+0000 7fee5496b640 10 monclient: monmap: 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.012+0000 7fee5496b640 1 librados: starting msgr at 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.012+0000 7fee5496b640 1 librados: starting objecter 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.012+0000 7fee5496b640 1 Processor -- start 2024-03-25T19:45:03.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.012+0000 7fee5496b640 1 -- start start 2024-03-25T19:45:03.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.012+0000 7fee5496b640 1 librados: setting wanted keys 2024-03-25T19:45:03.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.012+0000 7fee5496b640 1 librados: calling monclient init 2024-03-25T19:45:03.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.012+0000 7fee5496b640 10 monclient: init 2024-03-25T19:45:03.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 10 monclient(hunting): picked mon.noname-c con 0x7fee500a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 10 monclient(hunting): picked mon.noname-b con 0x7fee5013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 10 monclient(hunting): picked mon.noname-a con 0x7fee501430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee50137f10 con 0x7fee501430c0 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee501dc070 con 0x7fee5013f710 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee50137100 con 0x7fee500a98f0 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:03.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.013+0000 7fee5496b640 10 monclient(hunting): authenticate will time out at 5130.140127s 2024-03-25T19:45:03.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.025+0000 7fee47fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fee5013f710 0x7fee501db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:32956/0 (socket says 172.21.15.18:32956) 2024-03-25T19:45:03.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.025+0000 7fee47fff640 1 -- 172.21.15.18:0/3404790221 learned_addr learned my addr 172.21.15.18:0/3404790221 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:03.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.029+0000 7fee46ffd640 10 client.?.objecter ms_handle_connect 0x7fee501430c0 2024-03-25T19:45:03.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.029+0000 7fee46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.033+0000 7fee46ffd640 10 client.?.objecter ms_handle_connect 0x7fee5013f710 2024-03-25T19:45:03.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.033+0000 7fee46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.035+0000 7fee46ffd640 10 client.?.objecter ms_handle_connect 0x7fee500a98f0 2024-03-25T19:45:03.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.035+0000 7fee46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:03.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.042+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675040962 0 0) 0x7fee50137100 con 0x7fee500a98f0 2024-03-25T19:45:03.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.042+0000 7fee46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.042+0000 7fee46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.042+0000 7fee46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.042+0000 7fee46ffd640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:45:03.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.042+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee30004eb0 con 0x7fee500a98f0 2024-03-25T19:45:03.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.047+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284163718 0 0) 0x7fee501dc070 con 0x7fee5013f710 2024-03-25T19:45:03.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.047+0000 7fee46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.047+0000 7fee46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.047+0000 7fee46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.047+0000 7fee46ffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:45:03.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.047+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee50137100 con 0x7fee5013f710 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.055+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346834886 0 0) 0x7fee30004eb0 con 0x7fee500a98f0 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.055+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee501dc070 con 0x7fee500a98f0 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.055+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fee400018c0 con 0x7fee500a98f0 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.055+0000 7fee46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.055+0000 7fee46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.055+0000 7fee46ffd640 10 monclient(hunting): dump: 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:03.055 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:03.056 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:03.056 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:03.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.055+0000 7fee46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.058+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123406909 0 0) 0x7fee50137f10 con 0x7fee501430c0 2024-03-25T19:45:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.058+0000 7fee46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.058+0000 7fee46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.058+0000 7fee46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.058+0000 7fee46ffd640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:45:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.058+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee30004eb0 con 0x7fee501430c0 2024-03-25T19:45:03.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2922525663 0 0) 0x7fee501dc070 con 0x7fee500a98f0 2024-03-25T19:45:03.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 >> v1:172.21.15.53:6789/0 conn(0x7fee5013f710 legacy=0x7fee501db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 >> v1:172.21.15.18:6789/0 conn(0x7fee501430c0 legacy=0x7fee501db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:03.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 1 monclient: found mon.c 2024-03-25T19:45:03.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee501373e0 con 0x7fee500a98f0 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 10 monclient: _renew_subs 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee501358e0 con 0x7fee500a98f0 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 10 monclient: _renew_subs 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee50136150 con 0x7fee500a98f0 2024-03-25T19:45:03.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.069+0000 7fee5496b640 1 librados: init done 2024-03-25T19:45:03.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.070+0000 7fee5496b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:03.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.070+0000 7fee5496b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:03.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.070+0000 7fee5496b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:03.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.070+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee500627b0 con 0x7fee500a98f0 2024-03-25T19:45:03.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.080+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee400035f0 con 0x7fee500a98f0 2024-03-25T19:45:03.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.080+0000 7fee46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:03.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.080+0000 7fee4d59a640 4 set_mon_vals no callback set 2024-03-25T19:45:03.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.385+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fee40003a00 con 0x7fee500a98f0 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.385+0000 7fee46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.385+0000 7fee46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.385+0000 7fee46ffd640 10 monclient: dump: 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:03.385 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:03.386 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:03.386 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:03.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.386+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fee40003be0 con 0x7fee500a98f0 2024-03-25T19:45:03.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.386+0000 7fee47fff640 10 monclient: get_auth_request con 0x7fee3006de00 auth_method 0 2024-03-25T19:45:03.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.386+0000 7fee46ffd640 10 client.4385.objecter ms_handle_connect 0x7fee3006de00 2024-03-25T19:45:03.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.393+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fee40053610 con 0x7fee500a98f0 2024-03-25T19:45:03.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.393+0000 7fee46ffd640 10 client.4385.objecter ms_dispatch 0x7fee50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:03.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.393+0000 7fee46ffd640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:03.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.393+0000 7fee46ffd640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:03.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.393+0000 7fee46ffd640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-25T19:45:03.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.393+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee4004f120 con 0x7fee500a98f0 2024-03-25T19:45:03.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.393+0000 7fee46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:03.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.393+0000 7fee46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:03.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.649+0000 7fee5496b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:03.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.649+0000 7fee5496b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:03.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.649+0000 7fee5496b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:03.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.649+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee50135a70 con 0x7fee500a98f0 2024-03-25T19:45:03.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.679+0000 7fee46ffd640 1 -- 172.21.15.18:0/3404790221 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fee40051050 con 0x7fee500a98f0 2024-03-25T19:45:03.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.679+0000 7fee46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:03.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.679+0000 7fee46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:03.679 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:45:03.680 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 10 librados: watch_flush enter 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 10 librados: watch_flush exit 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 10 client.4385.objecter successfully canceled tick 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 >> v1:172.21.15.18:6800/1520000573 conn(0x7fee3006de00 legacy=0x7fee300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 10 monclient: shutdown 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:03.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 >> v1:172.21.15.18:6790/0 conn(0x7fee500a98f0 legacy=0x7fee50137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:03.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 shutdown_connections 2024-03-25T19:45:03.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 >> 172.21.15.18:0/3404790221 conn(0x7fee500b7ea0 msgr2=0x7fee5014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:03.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 shutdown_connections 2024-03-25T19:45:03.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 1 -- 172.21.15.18:0/3404790221 wait complete. 2024-03-25T19:45:03.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:03.682+0000 7fee5496b640 1 librados: shutdown 2024-03-25T19:45:03.694 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:07.483 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:35.013 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:07.541 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:35.072 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:07.585 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:35.115 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:07.598 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:35.128 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:07.607 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:35.138 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:07.616 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:04:35.112 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:07.620 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:04:35.117 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:07.645 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:35.176 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:07.660 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:04:35.156 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:10.501 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:10.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.617+0000 7f324d1cd640 10 monclient: get_monmap_and_config 2024-03-25T19:45:10.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.617+0000 7f324d1cd640 10 monclient: build_initial_monmap 2024-03-25T19:45:10.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.617+0000 7f324d1cd640 10 monclient: monmap: 2024-03-25T19:45:10.618 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:45:10.619470+0000 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:45:10.619470+0000 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:45:10.619 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:10.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.618+0000 7f324d1cd640 1 Processor -- start 2024-03-25T19:45:10.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.618+0000 7f324d1cd640 1 -- start start 2024-03-25T19:45:10.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.618+0000 7f324d1cd640 10 monclient: init 2024-03-25T19:45:10.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:10.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:10.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 10 monclient(hunting): picked mon.noname-b con 0x7f32480a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:10.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 10 monclient(hunting): picked mon.noname-c con 0x7f324813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:10.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 10 monclient(hunting): picked mon.noname-a con 0x7f32481430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:10.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3248142520 con 0x7f32481430c0 2024-03-25T19:45:10.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32480b28a0 con 0x7f324813f710 2024-03-25T19:45:10.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32480b2a30 con 0x7f32480a98f0 2024-03-25T19:45:10.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:10.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.619+0000 7f324d1cd640 10 monclient(hunting): authenticate will time out at 5137.747092s 2024-03-25T19:45:10.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.626+0000 7f324659c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f324813f710 0x7f32481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:56890/0 (socket says 172.21.15.18:56890) 2024-03-25T19:45:10.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.626+0000 7f324659c640 1 -- 172.21.15.18:0/1007754475 learned_addr learned my addr 172.21.15.18:0/1007754475 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:10.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709198311 0 0) 0x7f32480b2a30 con 0x7f32480a98f0 2024-03-25T19:45:10.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:45:10.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3230003520 con 0x7f32480a98f0 2024-03-25T19:45:10.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553034048 0 0) 0x7f32480b28a0 con 0x7f324813f710 2024-03-25T19:45:10.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:45:10.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.642+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32480b2a30 con 0x7f324813f710 2024-03-25T19:45:10.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 289818041 0 0) 0x7f32480b2a30 con 0x7f324813f710 2024-03-25T19:45:10.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 >> v1:172.21.15.18:6790/0 conn(0x7f32480a98f0 legacy=0x7f32480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 >> v1:172.21.15.18:6789/0 conn(0x7f32481430c0 legacy=0x7f324814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:45:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 1 monclient: found mon.noname-c 2024-03-25T19:45:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:45:10.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32480b44c0 con 0x7f324813f710 2024-03-25T19:45:10.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 10 monclient: _finish_auth 0 2024-03-25T19:45:10.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:45:10.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:10.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f323c001670 con 0x7f324813f710 2024-03-25T19:45:10.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:10.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f3245d9b640 10 monclient: dump: 2024-03-25T19:45:10.661 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:10.661 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:10.661 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:10.662 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:10.662 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.662 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:10.662 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:10.662 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:10.663 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:10.663 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:10.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f324d1cd640 5 monclient: authenticate success, global_id 4384 2024-03-25T19:45:10.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.656+0000 7f324d1cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:10.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.662+0000 7f3245d9b640 1 -- 172.21.15.18:0/1007754475 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f323c0017d0 con 0x7f324813f710 2024-03-25T19:45:10.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.662+0000 7f3245d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:10.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.662+0000 7f324d1cd640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:10.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.662+0000 7f324d1cd640 4 set_mon_vals no callback set 2024-03-25T19:45:10.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.663+0000 7f324d1cd640 10 monclient: shutdown 2024-03-25T19:45:10.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.663+0000 7f324d1cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:10.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.663+0000 7f324d1cd640 1 -- 172.21.15.18:0/1007754475 >> v1:172.21.15.53:6789/0 conn(0x7f324813f710 legacy=0x7f32481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.663+0000 7f324d1cd640 1 -- 172.21.15.18:0/1007754475 shutdown_connections 2024-03-25T19:45:10.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.663+0000 7f324d1cd640 1 -- 172.21.15.18:0/1007754475 >> 172.21.15.18:0/1007754475 conn(0x7f32480b7ea0 msgr2=0x7f32480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:10.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.663+0000 7f324d1cd640 1 -- 172.21.15.18:0/1007754475 shutdown_connections 2024-03-25T19:45:10.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.663+0000 7f324d1cd640 1 -- 172.21.15.18:0/1007754475 wait complete. 2024-03-25T19:45:10.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.664+0000 7f324d1cd640 10 monclient: build_initial_monmap 2024-03-25T19:45:10.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.664+0000 7f324d1cd640 10 monclient: monmap: 2024-03-25T19:45:10.670 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:10.670 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:10.670 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:45:10.671 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:45:10.671 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:10.671 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:45:10.671 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:10.671 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:45:10.671 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:45:10.672 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:10.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.664+0000 7f324d1cd640 1 librados: starting msgr at 2024-03-25T19:45:10.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.664+0000 7f324d1cd640 1 librados: starting objecter 2024-03-25T19:45:10.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 1 Processor -- start 2024-03-25T19:45:10.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 1 -- start start 2024-03-25T19:45:10.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 1 librados: setting wanted keys 2024-03-25T19:45:10.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 1 librados: calling monclient init 2024-03-25T19:45:10.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 10 monclient: init 2024-03-25T19:45:10.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:10.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:10.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 10 monclient(hunting): picked mon.noname-c con 0x7f32480a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:10.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 10 monclient(hunting): picked mon.noname-b con 0x7f324813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:10.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 10 monclient(hunting): picked mon.noname-a con 0x7f32481430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:10.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3248138f80 con 0x7f32481430c0 2024-03-25T19:45:10.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32480ab9e0 con 0x7f324813f710 2024-03-25T19:45:10.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32480abcc0 con 0x7f32480a98f0 2024-03-25T19:45:10.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:10.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.665+0000 7f324d1cd640 10 monclient(hunting): authenticate will time out at 5137.793721s 2024-03-25T19:45:10.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.670+0000 7f3246d9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f32480a98f0 0x7f32480ac3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:58844/0 (socket says 172.21.15.18:58844) 2024-03-25T19:45:10.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.670+0000 7f3246d9d640 1 -- 172.21.15.18:0/2094684886 learned_addr learned my addr 172.21.15.18:0/2094684886 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:10.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.670+0000 7f322f7fe640 10 client.?.objecter ms_handle_connect 0x7f32480a98f0 2024-03-25T19:45:10.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.670+0000 7f322f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.672+0000 7f322f7fe640 10 client.?.objecter ms_handle_connect 0x7f324813f710 2024-03-25T19:45:10.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.672+0000 7f322f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.676+0000 7f322f7fe640 10 client.?.objecter ms_handle_connect 0x7f32481430c0 2024-03-25T19:45:10.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.676+0000 7f322f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.676+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597020209 0 0) 0x7f32480ab9e0 con 0x7f324813f710 2024-03-25T19:45:10.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.676+0000 7f322f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.676+0000 7f322f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.676+0000 7f322f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.676+0000 7f322f7fe640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:45:10.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.676+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3234004eb0 con 0x7f324813f710 2024-03-25T19:45:10.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.680+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275131183 0 0) 0x7f32480abcc0 con 0x7f32480a98f0 2024-03-25T19:45:10.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.680+0000 7f322f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.680+0000 7f322f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.680+0000 7f322f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.680+0000 7f322f7fe640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:45:10.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.680+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32480ab9e0 con 0x7f32480a98f0 2024-03-25T19:45:10.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.688+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 846957330 0 0) 0x7f32480ab9e0 con 0x7f32480a98f0 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.688+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32480abcc0 con 0x7f32480a98f0 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.688+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3238001990 con 0x7f32480a98f0 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.688+0000 7f322f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.688+0000 7f322f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.688+0000 7f322f7fe640 10 monclient(hunting): dump: 2024-03-25T19:45:10.691 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:10.691 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:10.691 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:10.691 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:10.691 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.691 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:10.692 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:10.692 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:10.692 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:10.692 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:10.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.688+0000 7f322f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:10.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.700+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2530399214 0 0) 0x7f3234004eb0 con 0x7f324813f710 2024-03-25T19:45:10.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.700+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32480ab9e0 con 0x7f324813f710 2024-03-25T19:45:10.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.700+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f323c001c80 con 0x7f324813f710 2024-03-25T19:45:10.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.700+0000 7f322f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.700+0000 7f322f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:10.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.700+0000 7f322f7fe640 10 monclient(hunting): dump: 2024-03-25T19:45:10.702 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:10.703 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:10.703 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:10.703 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:10.703 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.703 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:10.703 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:10.703 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:10.704 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:10.704 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:10.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.701+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185594246 0 0) 0x7f3248138f80 con 0x7f32481430c0 2024-03-25T19:45:10.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.701+0000 7f322f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.701+0000 7f322f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.701+0000 7f322f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.701+0000 7f322f7fe640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:45:10.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.701+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3234004eb0 con 0x7f32481430c0 2024-03-25T19:45:10.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.709+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2859698434 0 0) 0x7f32480abcc0 con 0x7f32480a98f0 2024-03-25T19:45:10.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 >> v1:172.21.15.53:6789/0 conn(0x7f324813f710 legacy=0x7f32481386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 >> v1:172.21.15.18:6789/0 conn(0x7f32481430c0 legacy=0x7f3248136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:10.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 1 monclient: found mon.c 2024-03-25T19:45:10.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:10.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:10.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32480abfa0 con 0x7f32480a98f0 2024-03-25T19:45:10.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:10.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:10.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f322f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:10.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 5 monclient: authenticate success, global_id 4391 2024-03-25T19:45:10.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 10 monclient: _renew_subs 2024-03-25T19:45:10.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:10.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3248137bf0 con 0x7f32480a98f0 2024-03-25T19:45:10.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:10.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 10 monclient: _renew_subs 2024-03-25T19:45:10.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:10.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32481e3480 con 0x7f32480a98f0 2024-03-25T19:45:10.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.710+0000 7f324d1cd640 1 librados: init done 2024-03-25T19:45:10.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.711+0000 7f324d1cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:10.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.711+0000 7f324d1cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:10.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.711+0000 7f324d1cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:10.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:10.711+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32480627b0 con 0x7f32480a98f0 2024-03-25T19:45:11.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.032+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3238003610 con 0x7f32480a98f0 2024-03-25T19:45:11.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.032+0000 7f322f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:11.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.032+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f32380039e0 con 0x7f32480a98f0 2024-03-25T19:45:11.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.032+0000 7f322f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:11.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.032+0000 7f322f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:11.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.032+0000 7f322f7fe640 10 monclient: dump: 2024-03-25T19:45:11.033 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.032+0000 7f324559a640 4 set_mon_vals no callback set 2024-03-25T19:45:11.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.032+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f3238003c00 con 0x7f32480a98f0 2024-03-25T19:45:11.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.033+0000 7f324659c640 10 monclient: get_auth_request con 0x7f323406df50 auth_method 0 2024-03-25T19:45:11.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.033+0000 7f322f7fe640 10 client.4391.objecter ms_handle_connect 0x7f323406df50 2024-03-25T19:45:11.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.038+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f3238054800 con 0x7f32480a98f0 2024-03-25T19:45:11.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.038+0000 7f322f7fe640 10 client.4391.objecter ms_dispatch 0x7f3248000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:11.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.038+0000 7f322f7fe640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:11.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.038+0000 7f322f7fe640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:11.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.038+0000 7f322f7fe640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-25T19:45:11.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.038+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3238050310 con 0x7f32480a98f0 2024-03-25T19:45:11.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.038+0000 7f322f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:11.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.038+0000 7f322f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:11.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.271+0000 7f324d1cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:11.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.271+0000 7f324d1cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:11.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.271+0000 7f324d1cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:11.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.271+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3248074cc0 con 0x7f32480a98f0 2024-03-25T19:45:11.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.290+0000 7f322f7fe640 1 -- 172.21.15.18:0/2094684886 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f3238052240 con 0x7f32480a98f0 2024-03-25T19:45:11.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.290+0000 7f322f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:11.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.290+0000 7f322f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:11.291 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:45:11.291 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:11.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 10 librados: watch_flush enter 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 10 librados: watch_flush exit 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 10 client.4391.objecter successfully canceled tick 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 >> v1:172.21.15.18:6800/1520000573 conn(0x7f323406df50 legacy=0x7f3234070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 10 monclient: shutdown 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 >> v1:172.21.15.18:6790/0 conn(0x7f32480a98f0 legacy=0x7f32480ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 shutdown_connections 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.292+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 >> 172.21.15.18:0/2094684886 conn(0x7f32480b7ea0 msgr2=0x7f3248074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.293+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 shutdown_connections 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.293+0000 7f324d1cd640 1 -- 172.21.15.18:0/2094684886 wait complete. 2024-03-25T19:45:11.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:11.293+0000 7f324d1cd640 1 librados: shutdown 2024-03-25T19:45:11.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:18.113 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:18.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.217+0000 7f5b212c1640 10 monclient: get_monmap_and_config 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.217+0000 7f5b212c1640 10 monclient: build_initial_monmap 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.217+0000 7f5b212c1640 10 monclient: monmap: 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:45:18.218985+0000 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:45:18.218985+0000 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:45:18.218 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:45:18.219 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.217+0000 7f5b212c1640 1 Processor -- start 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.217+0000 7f5b212c1640 1 -- start start 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.217+0000 7f5b212c1640 10 monclient: init 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 10 monclient(hunting): picked mon.noname-b con 0x7f5b1c0705b0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 10 monclient(hunting): picked mon.noname-a con 0x7f5b1c1402d0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 10 monclient(hunting): picked mon.noname-c con 0x7f5b1c149ae0 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c0718c0 con 0x7f5b1c1402d0 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c0714c0 con 0x7f5b1c149ae0 2024-03-25T19:45:18.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c072b50 con 0x7f5b1c0705b0 2024-03-25T19:45:18.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:18.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.218+0000 7f5b212c1640 10 monclient(hunting): authenticate will time out at 5145.346513s 2024-03-25T19:45:18.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.221+0000 7f5b1b7fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5b1c149ae0 0x7f5b1c14bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:58474/0 (socket says 172.21.15.18:58474) 2024-03-25T19:45:18.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.221+0000 7f5b1b7fe640 1 -- 172.21.15.18:0/4153439634 learned_addr learned my addr 172.21.15.18:0/4153439634 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:18.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.229+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299779523 0 0) 0x7f5b1c0714c0 con 0x7f5b1c149ae0 2024-03-25T19:45:18.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.229+0000 7f5b19ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:18.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.229+0000 7f5b19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:18.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.229+0000 7f5b19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:18.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.229+0000 7f5b19ffb640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:45:18.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.229+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04003520 con 0x7f5b1c149ae0 2024-03-25T19:45:18.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2201557502 0 0) 0x7f5b04003520 con 0x7f5b1c149ae0 2024-03-25T19:45:18.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 >> v1:172.21.15.18:6790/0 conn(0x7f5b1c0705b0 legacy=0x7f5b1c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:45:18.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 >> v1:172.21.15.18:6789/0 conn(0x7f5b1c1402d0 legacy=0x7f5b1c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:18.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 1 monclient: found mon.noname-c 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c13dd10 con 0x7f5b1c149ae0 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5b14001a10 con 0x7f5b1c149ae0 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b19ffb640 10 monclient: dump: 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:18.236 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:18.237 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:18.237 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:18.237 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:18.237 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:18.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b212c1640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:45:18.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.234+0000 7f5b212c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.244+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b14001be0 con 0x7f5b1c149ae0 2024-03-25T19:45:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.244+0000 7f5b19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.244+0000 7f5b19ffb640 1 -- 172.21.15.18:0/4153439634 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5b14002030 con 0x7f5b1c149ae0 2024-03-25T19:45:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.244+0000 7f5b212c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.244+0000 7f5b212c1640 4 set_mon_vals no callback set 2024-03-25T19:45:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.244+0000 7f5b19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.244+0000 7f5b19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.244+0000 7f5b19ffb640 10 monclient: dump: 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:18.246 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.245+0000 7f5b212c1640 10 monclient: shutdown 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.245+0000 7f5b212c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.245+0000 7f5b212c1640 1 -- 172.21.15.18:0/4153439634 >> v1:172.21.15.53:6789/0 conn(0x7f5b1c149ae0 legacy=0x7f5b1c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.245+0000 7f5b212c1640 1 -- 172.21.15.18:0/4153439634 shutdown_connections 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.245+0000 7f5b212c1640 1 -- 172.21.15.18:0/4153439634 >> 172.21.15.18:0/4153439634 conn(0x7f5b1c0b5d10 msgr2=0x7f5b1c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.245+0000 7f5b212c1640 1 -- 172.21.15.18:0/4153439634 shutdown_connections 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.245+0000 7f5b212c1640 1 -- 172.21.15.18:0/4153439634 wait complete. 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.246+0000 7f5b212c1640 10 monclient: build_initial_monmap 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.246+0000 7f5b212c1640 10 monclient: monmap: 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:45:18.248 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.247+0000 7f5b212c1640 1 librados: starting msgr at 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.247+0000 7f5b212c1640 1 librados: starting objecter 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.247+0000 7f5b212c1640 1 Processor -- start 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.247+0000 7f5b212c1640 1 -- start start 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.247+0000 7f5b212c1640 1 librados: setting wanted keys 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.247+0000 7f5b212c1640 1 librados: calling monclient init 2024-03-25T19:45:18.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.247+0000 7f5b212c1640 10 monclient: init 2024-03-25T19:45:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 10 monclient(hunting): picked mon.noname-b con 0x7f5b1c0705b0 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 10 monclient(hunting): picked mon.noname-c con 0x7f5b1c1402d0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 10 monclient(hunting): picked mon.noname-a con 0x7f5b1c149ae0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c142bd0 con 0x7f5b1c149ae0 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c141cc0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c141fa0 con 0x7f5b1c1402d0 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.248+0000 7f5b212c1640 10 monclient(hunting): authenticate will time out at 5145.376553s 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.250+0000 7f5b1affd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5b1c0705b0 0x7f5b1c1424c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:58490/0 (socket says 172.21.15.18:58490) 2024-03-25T19:45:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.250+0000 7f5b1affd640 1 -- 172.21.15.18:0/2233630901 learned_addr learned my addr 172.21.15.18:0/2233630901 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:18.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.255+0000 7f5b037fe640 10 client.?.objecter ms_handle_connect 0x7f5b1c0705b0 2024-03-25T19:45:18.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.255+0000 7f5b037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:18.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.259+0000 7f5b037fe640 10 client.?.objecter ms_handle_connect 0x7f5b1c149ae0 2024-03-25T19:45:18.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.259+0000 7f5b037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.263+0000 7f5b037fe640 10 client.?.objecter ms_handle_connect 0x7f5b1c1402d0 2024-03-25T19:45:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.263+0000 7f5b037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.263+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416466962 0 0) 0x7f5b1c141cc0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.263+0000 7f5b037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.263+0000 7f5b037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.263+0000 7f5b037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.263+0000 7f5b037fe640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:45:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.263+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b08004eb0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.272+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656605132 0 0) 0x7f5b1c141fa0 con 0x7f5b1c1402d0 2024-03-25T19:45:18.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.273+0000 7f5b037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:18.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.273+0000 7f5b037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:18.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.273+0000 7f5b037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:18.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.273+0000 7f5b037fe640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:45:18.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.273+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c141cc0 con 0x7f5b1c1402d0 2024-03-25T19:45:18.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.276+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302289980 0 0) 0x7f5b1c142bd0 con 0x7f5b1c149ae0 2024-03-25T19:45:18.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.276+0000 7f5b037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:18.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.276+0000 7f5b037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:18.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.276+0000 7f5b037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:18.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.276+0000 7f5b037fe640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:45:18.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.276+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c141fa0 con 0x7f5b1c149ae0 2024-03-25T19:45:18.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.284+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180373241 0 0) 0x7f5b08004eb0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.284+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b1c142bd0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.284+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5b0c001960 con 0x7f5b1c0705b0 2024-03-25T19:45:18.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.284+0000 7f5b037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.284+0000 7f5b037fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.284+0000 7f5b037fe640 10 monclient(hunting): dump: 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:18.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.284+0000 7f5b037fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:18.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3612404184 0 0) 0x7f5b1c142bd0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 >> v1:172.21.15.18:6790/0 conn(0x7f5b1c1402d0 legacy=0x7f5b1c133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 >> v1:172.21.15.18:6789/0 conn(0x7f5b1c149ae0 legacy=0x7f5b1c1d8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:18.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 1 monclient: found mon.b 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c13dfd0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 5 monclient: authenticate success, global_id 4393 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 10 monclient: _renew_subs 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c13e280 con 0x7f5b1c0705b0 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 10 monclient: _renew_subs 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c134720 con 0x7f5b1c0705b0 2024-03-25T19:45:18.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.292+0000 7f5b212c1640 1 librados: init done 2024-03-25T19:45:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.293+0000 7f5b212c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.293+0000 7f5b212c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.293+0000 7f5b212c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.293+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c142130 con 0x7f5b1c0705b0 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.602+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b0c0035f0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.602+0000 7f5b037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.602+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5b0c003940 con 0x7f5b1c0705b0 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.602+0000 7f5b037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.602+0000 7f5b037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.602+0000 7f5b037fe640 10 monclient: dump: 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:18.603 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:18.604 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:18.604 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:18.604 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:18.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.602+0000 7f5b197fa640 4 set_mon_vals no callback set 2024-03-25T19:45:18.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.603+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f5b0c003b60 con 0x7f5b1c0705b0 2024-03-25T19:45:18.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.603+0000 7f5b1a7fc640 10 monclient: get_auth_request con 0x7f5b0806de20 auth_method 0 2024-03-25T19:45:18.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.604+0000 7f5b037fe640 10 client.4393.objecter ms_handle_connect 0x7f5b0806de20 2024-03-25T19:45:18.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.607+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f5b0c0547f0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.607+0000 7f5b037fe640 10 client.4393.objecter ms_dispatch 0x7f5b1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:18.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.607+0000 7f5b037fe640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:18.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.607+0000 7f5b037fe640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:18.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.607+0000 7f5b037fe640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T19:45:18.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.608+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b0c050300 con 0x7f5b1c0705b0 2024-03-25T19:45:18.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.608+0000 7f5b037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:18.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.608+0000 7f5b037fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:18.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.859+0000 7f5b212c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.859+0000 7f5b212c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.859+0000 7f5b212c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:18.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.859+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b1c0627b0 con 0x7f5b1c0705b0 2024-03-25T19:45:18.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.882+0000 7f5b037fe640 1 -- 172.21.15.18:0/2233630901 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f5b0c052230 con 0x7f5b1c0705b0 2024-03-25T19:45:18.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.882+0000 7f5b037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.882+0000 7f5b037fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:18.883 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:45:18.883 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:18.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.884+0000 7f5b212c1640 10 librados: watch_flush enter 2024-03-25T19:45:18.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.884+0000 7f5b212c1640 10 librados: watch_flush exit 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.884+0000 7f5b212c1640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.884+0000 7f5b212c1640 10 client.4393.objecter successfully canceled tick 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.884+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 >> v1:172.21.15.18:6800/1520000573 conn(0x7f5b0806de20 legacy=0x7f5b080702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.884+0000 7f5b212c1640 10 monclient: shutdown 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.884+0000 7f5b212c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.884+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 >> v1:172.21.15.53:6789/0 conn(0x7f5b1c0705b0 legacy=0x7f5b1c1424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.885+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 shutdown_connections 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.885+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 >> 172.21.15.18:0/2233630901 conn(0x7f5b1c0b5d10 msgr2=0x7f5b1c0b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:18.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.885+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 shutdown_connections 2024-03-25T19:45:18.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.885+0000 7f5b212c1640 1 -- 172.21.15.18:0/2233630901 wait complete. 2024-03-25T19:45:18.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:18.885+0000 7f5b212c1640 1 librados: shutdown 2024-03-25T19:45:18.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:25.704 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:25.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.818+0000 7fe3cbd7b640 10 monclient: get_monmap_and_config 2024-03-25T19:45:25.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.818+0000 7fe3cbd7b640 10 monclient: build_initial_monmap 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.818+0000 7fe3cbd7b640 10 monclient: monmap: 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:45:25.819740+0000 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:45:25.819740+0000 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:45:25.819 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.819+0000 7fe3cbd7b640 1 Processor -- start 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.819+0000 7fe3cbd7b640 1 -- start start 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.819+0000 7fe3cbd7b640 10 monclient: init 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c40b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c40b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c414be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40af940 con 0x7fe3c40b4d60 2024-03-25T19:45:25.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40b0a70 con 0x7fe3c414be90 2024-03-25T19:45:25.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40b0d50 con 0x7fe3c40b1730 2024-03-25T19:45:25.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:25.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.820+0000 7fe3cbd7b640 10 monclient(hunting): authenticate will time out at 5152.947578s 2024-03-25T19:45:25.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.826+0000 7fe3ca318640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe3c414be90 0x7fe3c414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:34202/0 (socket says 172.21.15.18:34202) 2024-03-25T19:45:25.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.826+0000 7fe3ca318640 1 -- 172.21.15.18:0/2519376418 learned_addr learned my addr 172.21.15.18:0/2519376418 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:25.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.832+0000 7fe3c8b15640 1 -- 172.21.15.18:0/2519376418 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594727562 0 0) 0x7fe3c40b0a70 con 0x7fe3c414be90 2024-03-25T19:45:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.833+0000 7fe3c8b15640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.833+0000 7fe3c8b15640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.833+0000 7fe3c8b15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.833+0000 7fe3c8b15640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:45:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.833+0000 7fe3c8b15640 1 -- 172.21.15.18:0/2519376418 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b0003520 con 0x7fe3c414be90 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 1 -- 172.21.15.18:0/2519376418 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3565952442 0 0) 0x7fe3b0003520 con 0x7fe3c414be90 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 1 -- 172.21.15.18:0/2519376418 >> v1:172.21.15.18:6790/0 conn(0x7fe3c40b1730 legacy=0x7fe3c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 1 -- 172.21.15.18:0/2519376418 >> v1:172.21.15.18:6789/0 conn(0x7fe3c40b4d60 legacy=0x7fe3c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 10 monclient: _finish_hunting 0 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 1 monclient: found mon.noname-c 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 1 -- 172.21.15.18:0/2519376418 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c40b1030 con 0x7fe3c414be90 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 10 monclient: _finish_auth 0 2024-03-25T19:45:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 10 monclient: _check_auth_tickets 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 1 -- 172.21.15.18:0/2519376418 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe3bc001a10 con 0x7fe3c414be90 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3c8b15640 10 monclient: dump: 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:25.843 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:25.844 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:25.844 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:25.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3cbd7b640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:45:25.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.842+0000 7fe3cbd7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:25.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.850+0000 7fe3c8b15640 1 -- 172.21.15.18:0/2519376418 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3bc001b70 con 0x7fe3c414be90 2024-03-25T19:45:25.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.850+0000 7fe3c8b15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:25.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.850+0000 7fe3cbd7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:25.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.850+0000 7fe3cbd7b640 4 set_mon_vals no callback set 2024-03-25T19:45:25.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.851+0000 7fe3cbd7b640 10 monclient: shutdown 2024-03-25T19:45:25.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.851+0000 7fe3cbd7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:25.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.851+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/2519376418 >> v1:172.21.15.53:6789/0 conn(0x7fe3c414be90 legacy=0x7fe3c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.851+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/2519376418 shutdown_connections 2024-03-25T19:45:25.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.851+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/2519376418 >> 172.21.15.18:0/2519376418 conn(0x7fe3c40617a0 msgr2=0x7fe3c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:25.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.851+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/2519376418 shutdown_connections 2024-03-25T19:45:25.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.851+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/2519376418 wait complete. 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 10 monclient: build_initial_monmap 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 10 monclient: monmap: 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:45:25.852 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 1 librados: starting msgr at 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 1 librados: starting objecter 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 1 Processor -- start 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 1 -- start start 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 1 librados: setting wanted keys 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 1 librados: calling monclient init 2024-03-25T19:45:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.852+0000 7fe3cbd7b640 10 monclient: init 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c40b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c40b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c414be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c41d7b80 con 0x7fe3c40b4d60 2024-03-25T19:45:25.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c41431b0 con 0x7fe3c414be90 2024-03-25T19:45:25.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c41d77a0 con 0x7fe3c40b1730 2024-03-25T19:45:25.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:25.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.853+0000 7fe3cbd7b640 10 monclient(hunting): authenticate will time out at 5152.980217s 2024-03-25T19:45:25.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.862+0000 7fe3ca318640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe3c414be90 0x7fe3c4134d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:34240/0 (socket says 172.21.15.18:34240) 2024-03-25T19:45:25.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.862+0000 7fe3ca318640 1 -- 172.21.15.18:0/3692338024 learned_addr learned my addr 172.21.15.18:0/3692338024 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:25.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.865+0000 7fe3ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe3c414be90 2024-03-25T19:45:25.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.865+0000 7fe3ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.872+0000 7fe3ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe3c40b1730 2024-03-25T19:45:25.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.872+0000 7fe3ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.875+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840106247 0 0) 0x7fe3c41d77a0 con 0x7fe3c40b1730 2024-03-25T19:45:25.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.875+0000 7fe3ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.875+0000 7fe3ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.875+0000 7fe3ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.875+0000 7fe3ba7fc640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:45:25.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.875+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a8004eb0 con 0x7fe3c40b1730 2024-03-25T19:45:25.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.877+0000 7fe3ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe3c40b4d60 2024-03-25T19:45:25.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.877+0000 7fe3ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.888+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52804437 0 0) 0x7fe3c41431b0 con 0x7fe3c414be90 2024-03-25T19:45:25.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.888+0000 7fe3ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.888+0000 7fe3ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.888+0000 7fe3ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.888+0000 7fe3ba7fc640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:45:25.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.888+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c41d77a0 con 0x7fe3c414be90 2024-03-25T19:45:25.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.891+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178341181 0 0) 0x7fe3c41d7b80 con 0x7fe3c40b4d60 2024-03-25T19:45:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.891+0000 7fe3ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.891+0000 7fe3ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.891+0000 7fe3ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.891+0000 7fe3ba7fc640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:45:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.891+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c41431b0 con 0x7fe3c40b4d60 2024-03-25T19:45:25.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.892+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090411166 0 0) 0x7fe3a8004eb0 con 0x7fe3c40b1730 2024-03-25T19:45:25.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.892+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3c41d7b80 con 0x7fe3c40b1730 2024-03-25T19:45:25.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.892+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe3c0001930 con 0x7fe3c40b1730 2024-03-25T19:45:25.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.892+0000 7fe3ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.892+0000 7fe3ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.892+0000 7fe3ba7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.892+0000 7fe3ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:25.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.903+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 428490631 0 0) 0x7fe3c41d77a0 con 0x7fe3c414be90 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.904+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3a8004eb0 con 0x7fe3c414be90 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.904+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe3bc001e80 con 0x7fe3c414be90 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.904+0000 7fe3ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.904+0000 7fe3ba7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.904+0000 7fe3ba7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:25.904 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:25.905 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4005124672 0 0) 0x7fe3c41d7b80 con 0x7fe3c40b1730 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 >> v1:172.21.15.53:6789/0 conn(0x7fe3c414be90 legacy=0x7fe3c4134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 >> v1:172.21.15.18:6789/0 conn(0x7fe3c40b4d60 legacy=0x7fe3c41377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 1 monclient: found mon.c 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c41d7930 con 0x7fe3c40b1730 2024-03-25T19:45:25.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 5 monclient: authenticate success, global_id 4400 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 10 monclient: _renew_subs 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c41427a0 con 0x7fe3c40b1730 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 10 monclient: _renew_subs 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c4135f50 con 0x7fe3c40b1730 2024-03-25T19:45:25.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.909+0000 7fe3cbd7b640 1 librados: init done 2024-03-25T19:45:25.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.910+0000 7fe3cbd7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.910+0000 7fe3cbd7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.910+0000 7fe3cbd7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:25.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.910+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c41371d0 con 0x7fe3c40b1730 2024-03-25T19:45:25.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.915+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3c00035f0 con 0x7fe3c40b1730 2024-03-25T19:45:25.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.915+0000 7fe3ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:25.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:25.915+0000 7fe3bbfff640 4 set_mon_vals no callback set 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.230+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe3c0003980 con 0x7fe3c40b1730 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.230+0000 7fe3ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.230+0000 7fe3ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.230+0000 7fe3ba7fc640 10 monclient: dump: 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:26.230 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:26.231 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:26.231 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:26.231 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:26.231 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:26.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.231+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fe3c0003b60 con 0x7fe3c40b1730 2024-03-25T19:45:26.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.231+0000 7fe3c9316640 10 monclient: get_auth_request con 0x7fe3a806df20 auth_method 0 2024-03-25T19:45:26.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.232+0000 7fe3ba7fc640 10 client.4400.objecter ms_handle_connect 0x7fe3a806df20 2024-03-25T19:45:26.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.236+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fe3c0053680 con 0x7fe3c40b1730 2024-03-25T19:45:26.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.236+0000 7fe3ba7fc640 10 client.4400.objecter ms_dispatch 0x7fe3c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:26.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.236+0000 7fe3ba7fc640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:26.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.236+0000 7fe3ba7fc640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:26.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.236+0000 7fe3ba7fc640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-25T19:45:26.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.236+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3c004f190 con 0x7fe3c40b1730 2024-03-25T19:45:26.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.236+0000 7fe3ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:26.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.236+0000 7fe3ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:26.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.487+0000 7fe3cbd7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:26.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.487+0000 7fe3cbd7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:26.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.487+0000 7fe3cbd7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:26.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.487+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c40627b0 con 0x7fe3c40b1730 2024-03-25T19:45:26.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.534+0000 7fe3ba7fc640 1 -- 172.21.15.18:0/3692338024 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fe3c00510c0 con 0x7fe3c40b1730 2024-03-25T19:45:26.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.534+0000 7fe3ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:26.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.534+0000 7fe3ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:26.534 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:45:26.534 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:26.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 10 librados: watch_flush enter 2024-03-25T19:45:26.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 10 librados: watch_flush exit 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 10 client.4400.objecter successfully canceled tick 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 >> v1:172.21.15.18:6800/1520000573 conn(0x7fe3a806df20 legacy=0x7fe3a80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 10 monclient: shutdown 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 >> v1:172.21.15.18:6790/0 conn(0x7fe3c40b1730 legacy=0x7fe3c41d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 shutdown_connections 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 >> 172.21.15.18:0/3692338024 conn(0x7fe3c40617a0 msgr2=0x7fe3c413c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.536+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 shutdown_connections 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.537+0000 7fe3cbd7b640 1 -- 172.21.15.18:0/3692338024 wait complete. 2024-03-25T19:45:26.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:26.537+0000 7fe3cbd7b640 1 librados: shutdown 2024-03-25T19:45:26.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:27.716 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:55.246 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:27.723 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:55.254 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:27.754 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:55.285 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:27.781 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:55.312 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:27.788 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:04:55.284 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:27.794 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:55.324 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:27.794 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:04:55.290 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:27.827 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:04:55.324 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:27.832 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:55.363 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:33.357 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.471+0000 7f90f2606640 10 monclient: get_monmap_and_config 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.471+0000 7f90f2606640 10 monclient: build_initial_monmap 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.471+0000 7f90f2606640 10 monclient: monmap: 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:45:33.472179+0000 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:45:33.472179+0000 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:33.471 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:33.472 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:45:33.472 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:45:33.472 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:33.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 1 Processor -- start 2024-03-25T19:45:33.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 1 -- start start 2024-03-25T19:45:33.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 10 monclient: init 2024-03-25T19:45:33.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:33.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:33.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 10 monclient(hunting): picked mon.noname-c con 0x7f90ec0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:33.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 10 monclient(hunting): picked mon.noname-a con 0x7f90ec13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:33.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 10 monclient(hunting): picked mon.noname-b con 0x7f90ec1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:33.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec142520 con 0x7f90ec13f710 2024-03-25T19:45:33.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec0b28a0 con 0x7f90ec0a98f0 2024-03-25T19:45:33.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec0b2a30 con 0x7f90ec1430c0 2024-03-25T19:45:33.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:33.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.472+0000 7f90f2606640 10 monclient(hunting): authenticate will time out at 5160.599708s 2024-03-25T19:45:33.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.480+0000 7f90eb7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f90ec13f710 0x7f90ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:39216/0 (socket says 172.21.15.18:39216) 2024-03-25T19:45:33.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.480+0000 7f90eb7fe640 1 -- 172.21.15.18:0/4259082876 learned_addr learned my addr 172.21.15.18:0/4259082876 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:33.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.498+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904189499 0 0) 0x7f90ec0b2a30 con 0x7f90ec1430c0 2024-03-25T19:45:33.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.498+0000 7f90eaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.498+0000 7f90eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.498+0000 7f90eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.498+0000 7f90eaffd640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:45:33.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.498+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d4003520 con 0x7f90ec1430c0 2024-03-25T19:45:33.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4216160744 0 0) 0x7f90d4003520 con 0x7f90ec1430c0 2024-03-25T19:45:33.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 >> v1:172.21.15.53:6789/0 conn(0x7f90ec0a98f0 legacy=0x7f90ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 >> v1:172.21.15.18:6789/0 conn(0x7f90ec13f710 legacy=0x7f90ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 1 monclient: found mon.noname-b 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec0b44c0 con 0x7f90ec1430c0 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f90e4001a10 con 0x7f90ec1430c0 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.502+0000 7f90eaffd640 10 monclient: dump: 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:33.503 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.503+0000 7f90f2606640 5 monclient: authenticate success, global_id 4403 2024-03-25T19:45:33.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.503+0000 7f90f2606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.518+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90e4001b70 con 0x7f90ec1430c0 2024-03-25T19:45:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.518+0000 7f90eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.518+0000 7f90eaffd640 1 -- 172.21.15.18:0/4259082876 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f90e4001f80 con 0x7f90ec1430c0 2024-03-25T19:45:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.518+0000 7f90f2606640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.518+0000 7f90f2606640 4 set_mon_vals no callback set 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.518+0000 7f90eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.518+0000 7f90eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.518+0000 7f90eaffd640 10 monclient: dump: 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.519+0000 7f90f2606640 10 monclient: shutdown 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.519+0000 7f90f2606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.519+0000 7f90f2606640 1 -- 172.21.15.18:0/4259082876 >> v1:172.21.15.18:6790/0 conn(0x7f90ec1430c0 legacy=0x7f90ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.519+0000 7f90f2606640 1 -- 172.21.15.18:0/4259082876 shutdown_connections 2024-03-25T19:45:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.519+0000 7f90f2606640 1 -- 172.21.15.18:0/4259082876 >> 172.21.15.18:0/4259082876 conn(0x7f90ec0b7ea0 msgr2=0x7f90ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.519+0000 7f90f2606640 1 -- 172.21.15.18:0/4259082876 shutdown_connections 2024-03-25T19:45:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.519+0000 7f90f2606640 1 -- 172.21.15.18:0/4259082876 wait complete. 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.520+0000 7f90f2606640 10 monclient: build_initial_monmap 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.520+0000 7f90f2606640 10 monclient: monmap: 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:45:33.522 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.520+0000 7f90f2606640 1 librados: starting msgr at 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.520+0000 7f90f2606640 1 librados: starting objecter 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 1 Processor -- start 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 1 -- start start 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 1 librados: setting wanted keys 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 1 librados: calling monclient init 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 10 monclient: init 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 10 monclient(hunting): picked mon.noname-a con 0x7f90ec0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 10 monclient(hunting): picked mon.noname-b con 0x7f90ec13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:33.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 10 monclient(hunting): picked mon.noname-c con 0x7f90ec1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:33.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec137c60 con 0x7f90ec0a98f0 2024-03-25T19:45:33.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec139160 con 0x7f90ec13f710 2024-03-25T19:45:33.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ec1380f0 con 0x7f90ec1430c0 2024-03-25T19:45:33.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:33.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.521+0000 7f90f2606640 10 monclient(hunting): authenticate will time out at 5160.648647s 2024-03-25T19:45:33.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.531+0000 7f90eb7fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f90ec13f710 0x7f90ec133070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:38708/0 (socket says 172.21.15.18:38708) 2024-03-25T19:45:33.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.531+0000 7f90eb7fe640 1 -- 172.21.15.18:0/57178522 learned_addr learned my addr 172.21.15.18:0/57178522 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.532+0000 7f90e8ff9640 10 client.?.objecter ms_handle_connect 0x7f90ec13f710 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.532+0000 7f90e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.533+0000 7f90e8ff9640 10 client.?.objecter ms_handle_connect 0x7f90ec1430c0 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.533+0000 7f90e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.543+0000 7f90e8ff9640 10 client.?.objecter ms_handle_connect 0x7f90ec0a98f0 2024-03-25T19:45:33.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.543+0000 7f90e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.547+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878302249 0 0) 0x7f90ec139160 con 0x7f90ec13f710 2024-03-25T19:45:33.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.547+0000 7f90e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.547+0000 7f90e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.547+0000 7f90e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.547+0000 7f90e8ff9640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:45:33.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.547+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d8004eb0 con 0x7f90ec13f710 2024-03-25T19:45:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.548+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860937369 0 0) 0x7f90ec1380f0 con 0x7f90ec1430c0 2024-03-25T19:45:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.548+0000 7f90e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.548+0000 7f90e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.548+0000 7f90e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.548+0000 7f90e8ff9640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:45:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.548+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90ec139160 con 0x7f90ec1430c0 2024-03-25T19:45:33.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.553+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704399687 0 0) 0x7f90ec137c60 con 0x7f90ec0a98f0 2024-03-25T19:45:33.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.553+0000 7f90e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.553+0000 7f90e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.553+0000 7f90e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.553+0000 7f90e8ff9640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:45:33.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.553+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90ec1380f0 con 0x7f90ec0a98f0 2024-03-25T19:45:33.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.564+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3613922149 0 0) 0x7f90ec139160 con 0x7f90ec1430c0 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.564+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90ec137c60 con 0x7f90ec1430c0 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.564+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f90e4001ec0 con 0x7f90ec1430c0 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.564+0000 7f90e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.564+0000 7f90e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.564+0000 7f90e8ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:33.564 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:33.565 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:33.565 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:33.565 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:33.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.564+0000 7f90e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:33.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.575+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3431477923 0 0) 0x7f90d8004eb0 con 0x7f90ec13f710 2024-03-25T19:45:33.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.575+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90ec139160 con 0x7f90ec13f710 2024-03-25T19:45:33.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.575+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f90e00018b0 con 0x7f90ec13f710 2024-03-25T19:45:33.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.575+0000 7f90e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.575+0000 7f90e8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.575+0000 7f90e8ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:33.576 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:33.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.577+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 297730537 0 0) 0x7f90ec1380f0 con 0x7f90ec0a98f0 2024-03-25T19:45:33.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.577+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90d8004eb0 con 0x7f90ec0a98f0 2024-03-25T19:45:33.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.577+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f90dc001860 con 0x7f90ec0a98f0 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.577+0000 7f90e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.577+0000 7f90e8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.577+0000 7f90e8ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:33.578 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:33.579 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:33.579 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.581+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2735069348 0 0) 0x7f90ec137c60 con 0x7f90ec1430c0 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 >> v1:172.21.15.53:6789/0 conn(0x7f90ec13f710 legacy=0x7f90ec133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 >> v1:172.21.15.18:6789/0 conn(0x7f90ec0a98f0 legacy=0x7f90ec1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 1 monclient: found mon.c 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec1397f0 con 0x7f90ec1430c0 2024-03-25T19:45:33.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 5 monclient: authenticate success, global_id 4406 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 10 monclient: _renew_subs 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec1db280 con 0x7f90ec1430c0 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 10 monclient: _renew_subs 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ec133960 con 0x7f90ec1430c0 2024-03-25T19:45:33.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.582+0000 7f90f2606640 1 librados: init done 2024-03-25T19:45:33.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.583+0000 7f90f2606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.583+0000 7f90f2606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.583+0000 7f90f2606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:33.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.583+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ec1db5d0 con 0x7f90ec1430c0 2024-03-25T19:45:33.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.908+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90e40020c0 con 0x7f90ec1430c0 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.908+0000 7f90e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.908+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f90e4003dc0 con 0x7f90ec1430c0 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.908+0000 7f90e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.908+0000 7f90e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.908+0000 7f90e8ff9640 10 monclient: dump: 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:33.908 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:33.909 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:33.909 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:33.909 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:33.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.908+0000 7f90ea7fc640 4 set_mon_vals no callback set 2024-03-25T19:45:33.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.909+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f90e4019d90 con 0x7f90ec1430c0 2024-03-25T19:45:33.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.909+0000 7f90ebfff640 10 monclient: get_auth_request con 0x7f90d806e0f0 auth_method 0 2024-03-25T19:45:33.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.909+0000 7f90e8ff9640 10 client.4406.objecter ms_handle_connect 0x7f90d806e0f0 2024-03-25T19:45:33.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.915+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f90e40538f0 con 0x7f90ec1430c0 2024-03-25T19:45:33.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.915+0000 7f90e8ff9640 10 client.4406.objecter ms_dispatch 0x7f90ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:33.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.915+0000 7f90e8ff9640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:33.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.915+0000 7f90e8ff9640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:33.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.915+0000 7f90e8ff9640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-25T19:45:33.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.915+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90e404f230 con 0x7f90ec1430c0 2024-03-25T19:45:33.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.915+0000 7f90e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:33.915+0000 7f90e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:34.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.147+0000 7f90f2606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:34.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.147+0000 7f90f2606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:34.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.147+0000 7f90f2606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:34.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.147+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90ec137e10 con 0x7f90ec1430c0 2024-03-25T19:45:34.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.172+0000 7f90e8ff9640 1 -- 172.21.15.18:0/57178522 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f90ec137e10 con 0x7f90ec1430c0 2024-03-25T19:45:34.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.172+0000 7f90e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:34.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.172+0000 7f90e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:34.173 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:45:34.173 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:34.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 10 librados: watch_flush enter 2024-03-25T19:45:34.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 10 librados: watch_flush exit 2024-03-25T19:45:34.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 10 client.4406.objecter successfully canceled tick 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 >> v1:172.21.15.18:6800/1520000573 conn(0x7f90d806e0f0 legacy=0x7f90d80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 10 monclient: shutdown 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 >> v1:172.21.15.18:6790/0 conn(0x7f90ec1430c0 legacy=0x7f90ec1e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 shutdown_connections 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 >> 172.21.15.18:0/57178522 conn(0x7f90ec0b7ea0 msgr2=0x7f90ec13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 shutdown_connections 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 1 -- 172.21.15.18:0/57178522 wait complete. 2024-03-25T19:45:34.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:34.175+0000 7f90f2606640 1 librados: shutdown 2024-03-25T19:45:34.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:40.995 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:41.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.114+0000 7fc40d30d640 10 monclient: get_monmap_and_config 2024-03-25T19:45:41.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.114+0000 7fc40d30d640 10 monclient: build_initial_monmap 2024-03-25T19:45:41.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.114+0000 7fc40d30d640 10 monclient: monmap: 2024-03-25T19:45:41.114 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:41.114 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:41.114 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:45:41.115171+0000 2024-03-25T19:45:41.114 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:45:41.115171+0000 2024-03-25T19:45:41.114 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:41.115 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:41.115 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:41.115 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:45:41.115 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:45:41.115 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:41.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 1 Processor -- start 2024-03-25T19:45:41.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 1 -- start start 2024-03-25T19:45:41.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 10 monclient: init 2024-03-25T19:45:41.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:41.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:41.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4080a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:41.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 10 monclient(hunting): picked mon.noname-c con 0x7fc40813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:41.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 10 monclient(hunting): picked mon.noname-b con 0x7fc4081430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:41.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc408142520 con 0x7fc4080a98f0 2024-03-25T19:45:41.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4080b28a0 con 0x7fc40813f710 2024-03-25T19:45:41.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4080b2a30 con 0x7fc4081430c0 2024-03-25T19:45:41.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:41.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.115+0000 7fc40d30d640 10 monclient(hunting): authenticate will time out at 5168.242576s 2024-03-25T19:45:41.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.123+0000 7fc4077fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc4081430c0 0x7fc40814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:32834/0 (socket says 172.21.15.18:32834) 2024-03-25T19:45:41.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.123+0000 7fc4077fe640 1 -- 172.21.15.18:0/3239880968 learned_addr learned my addr 172.21.15.18:0/3239880968 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.143+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680798554 0 0) 0x7fc408142520 con 0x7fc4080a98f0 2024-03-25T19:45:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.143+0000 7fc405ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.143+0000 7fc405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.143+0000 7fc405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.143+0000 7fc405ffb640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:45:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.143+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f0003520 con 0x7fc4080a98f0 2024-03-25T19:45:41.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.148+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805722429 0 0) 0x7fc4080b2a30 con 0x7fc4081430c0 2024-03-25T19:45:41.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.148+0000 7fc405ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:41.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.148+0000 7fc405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:41.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.148+0000 7fc405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:41.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.148+0000 7fc405ffb640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:45:41.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.148+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc408142520 con 0x7fc4081430c0 2024-03-25T19:45:41.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.150+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191057174 0 0) 0x7fc4080b28a0 con 0x7fc40813f710 2024-03-25T19:45:41.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.150+0000 7fc405ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:41.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.150+0000 7fc405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:41.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.150+0000 7fc405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:41.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.150+0000 7fc405ffb640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:45:41.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.150+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4080b2a30 con 0x7fc40813f710 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.163+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2973852049 0 0) 0x7fc3f0003520 con 0x7fc4080a98f0 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.163+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 >> v1:172.21.15.18:6790/0 conn(0x7fc4081430c0 legacy=0x7fc40814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.163+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 >> v1:172.21.15.53:6789/0 conn(0x7fc40813f710 legacy=0x7fc4081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 1 monclient: found mon.noname-a 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4080b44c0 con 0x7fc4080a98f0 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc3f8001690 con 0x7fc4080a98f0 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc40d30d640 5 monclient: authenticate success, global_id 4440 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc40d30d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:41.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc405ffb640 10 monclient: dump: 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:41.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.164+0000 7fc40d30d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.181+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3f80017f0 con 0x7fc4080a98f0 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.181+0000 7fc405ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.181+0000 7fc405ffb640 1 -- 172.21.15.18:0/3239880968 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc3f8001c00 con 0x7fc4080a98f0 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.181+0000 7fc405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.181+0000 7fc405ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.181+0000 7fc405ffb640 10 monclient: dump: 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:41.181 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:41.182 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:41.182 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:41.182 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:41.182 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:41.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.181+0000 7fc40d30d640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:41.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.181+0000 7fc40d30d640 4 set_mon_vals no callback set 2024-03-25T19:45:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.182+0000 7fc40d30d640 10 monclient: shutdown 2024-03-25T19:45:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.182+0000 7fc40d30d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.182+0000 7fc40d30d640 1 -- 172.21.15.18:0/3239880968 >> v1:172.21.15.18:6789/0 conn(0x7fc4080a98f0 legacy=0x7fc4080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.182+0000 7fc40d30d640 1 -- 172.21.15.18:0/3239880968 shutdown_connections 2024-03-25T19:45:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.182+0000 7fc40d30d640 1 -- 172.21.15.18:0/3239880968 >> 172.21.15.18:0/3239880968 conn(0x7fc4080b7ea0 msgr2=0x7fc4080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.182+0000 7fc40d30d640 1 -- 172.21.15.18:0/3239880968 shutdown_connections 2024-03-25T19:45:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.182+0000 7fc40d30d640 1 -- 172.21.15.18:0/3239880968 wait complete. 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.183+0000 7fc40d30d640 10 monclient: build_initial_monmap 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.183+0000 7fc40d30d640 10 monclient: monmap: 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.184+0000 7fc40d30d640 1 librados: starting msgr at 2024-03-25T19:45:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.184+0000 7fc40d30d640 1 librados: starting objecter 2024-03-25T19:45:41.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.184+0000 7fc40d30d640 1 Processor -- start 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.184+0000 7fc40d30d640 1 -- start start 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.184+0000 7fc40d30d640 1 librados: setting wanted keys 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.184+0000 7fc40d30d640 1 librados: calling monclient init 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.184+0000 7fc40d30d640 10 monclient: init 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 10 monclient(hunting): picked mon.noname-b con 0x7fc4080a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 10 monclient(hunting): picked mon.noname-a con 0x7fc40813f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4081430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4081369b0 con 0x7fc40813f710 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4081366b0 con 0x7fc4080a98f0 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc408133ad0 con 0x7fc4081430c0 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:41.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.185+0000 7fc40d30d640 10 monclient(hunting): authenticate will time out at 5168.312415s 2024-03-25T19:45:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.198+0000 7fc4067fc640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc40813f710 0x7fc408132cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47494/0 (socket says 172.21.15.18:47494) 2024-03-25T19:45:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.198+0000 7fc4067fc640 1 -- 172.21.15.18:0/880380009 learned_addr learned my addr 172.21.15.18:0/880380009 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:41.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.199+0000 7fc3ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc40813f710 2024-03-25T19:45:41.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.199+0000 7fc3ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:41.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.201+0000 7fc3ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc4081430c0 2024-03-25T19:45:41.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.201+0000 7fc3ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:41.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.209+0000 7fc3ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc4080a98f0 2024-03-25T19:45:41.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.209+0000 7fc3ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:41.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.221+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974198231 0 0) 0x7fc4081366b0 con 0x7fc4080a98f0 2024-03-25T19:45:41.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.221+0000 7fc3ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:41.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.221+0000 7fc3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:41.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.221+0000 7fc3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:41.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.221+0000 7fc3ef7fe640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:45:41.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.221+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f4004eb0 con 0x7fc4080a98f0 2024-03-25T19:45:41.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.222+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379186175 0 0) 0x7fc408133ad0 con 0x7fc4081430c0 2024-03-25T19:45:41.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.222+0000 7fc3ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:41.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.222+0000 7fc3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:41.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.222+0000 7fc3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:41.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.222+0000 7fc3ef7fe640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:45:41.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.222+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4081366b0 con 0x7fc4081430c0 2024-03-25T19:45:41.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.224+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861287018 0 0) 0x7fc4081369b0 con 0x7fc40813f710 2024-03-25T19:45:41.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.224+0000 7fc3ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:41.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.224+0000 7fc3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:41.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.224+0000 7fc3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:41.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.224+0000 7fc3ef7fe640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:45:41.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.224+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc408133ad0 con 0x7fc40813f710 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.238+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 337377117 0 0) 0x7fc3f4004eb0 con 0x7fc4080a98f0 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.238+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4081369b0 con 0x7fc4080a98f0 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.239+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc3f8001b00 con 0x7fc4080a98f0 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.239+0000 7fc3ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.239+0000 7fc3ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.239+0000 7fc3ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.239+0000 7fc3ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.243+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3406018949 0 0) 0x7fc4081366b0 con 0x7fc4081430c0 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.243+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3f4004eb0 con 0x7fc4081430c0 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.243+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc400001b50 con 0x7fc4081430c0 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.243+0000 7fc3ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.243+0000 7fc3ef7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.243+0000 7fc3ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:41.243 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:41.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.253+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490979516 0 0) 0x7fc408133ad0 con 0x7fc40813f710 2024-03-25T19:45:41.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.253+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4081366b0 con 0x7fc40813f710 2024-03-25T19:45:41.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.253+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc3fc0019b0 con 0x7fc40813f710 2024-03-25T19:45:41.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.253+0000 7fc3ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:41.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.253+0000 7fc3ef7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:41.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.253+0000 7fc3ef7fe640 10 monclient(hunting): dump: 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:41.254 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:41.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2812776704 0 0) 0x7fc3f4004eb0 con 0x7fc4081430c0 2024-03-25T19:45:41.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 >> v1:172.21.15.53:6789/0 conn(0x7fc4080a98f0 legacy=0x7fc4081371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 >> v1:172.21.15.18:6789/0 conn(0x7fc40813f710 legacy=0x7fc408132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:41.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 1 monclient: found mon.c 2024-03-25T19:45:41.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:41.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc408136e40 con 0x7fc4081430c0 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc3ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 5 monclient: authenticate success, global_id 4412 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 10 monclient: _renew_subs 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4081378f0 con 0x7fc4081430c0 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 10 monclient: _renew_subs 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4080b1480 con 0x7fc4081430c0 2024-03-25T19:45:41.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.263+0000 7fc40d30d640 1 librados: init done 2024-03-25T19:45:41.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.264+0000 7fc40d30d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:41.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.264+0000 7fc40d30d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:41.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.264+0000 7fc40d30d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:41.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.264+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc408137030 con 0x7fc4081430c0 2024-03-25T19:45:41.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.625+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc400002060 con 0x7fc4081430c0 2024-03-25T19:45:41.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.625+0000 7fc3ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:41.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.625+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc400004d00 con 0x7fc4081430c0 2024-03-25T19:45:41.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.625+0000 7fc3ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:41.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.625+0000 7fc3ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:41.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.625+0000 7fc3ef7fe640 10 monclient: dump: 2024-03-25T19:45:41.625 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:41.625 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.625+0000 7fc4057fa640 4 set_mon_vals no callback set 2024-03-25T19:45:41.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.625+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fc40001ad10 con 0x7fc4081430c0 2024-03-25T19:45:41.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.626+0000 7fc406ffd640 10 monclient: get_auth_request con 0x7fc3f406e080 auth_method 0 2024-03-25T19:45:41.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.626+0000 7fc3ef7fe640 10 client.4412.objecter ms_handle_connect 0x7fc3f406e080 2024-03-25T19:45:41.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.632+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fc400054aa0 con 0x7fc4081430c0 2024-03-25T19:45:41.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.632+0000 7fc3ef7fe640 10 client.4412.objecter ms_dispatch 0x7fc408000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:41.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.632+0000 7fc3ef7fe640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:41.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.632+0000 7fc3ef7fe640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:41.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.632+0000 7fc3ef7fe640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T19:45:41.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.632+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4000503e0 con 0x7fc4081430c0 2024-03-25T19:45:41.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.632+0000 7fc3ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:41.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.632+0000 7fc3ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:41.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.890+0000 7fc40d30d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.890+0000 7fc40d30d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.890+0000 7fc40d30d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:45:41.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.890+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc408136b60 con 0x7fc4081430c0 2024-03-25T19:45:41.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.934+0000 7fc3ef7fe640 1 -- 172.21.15.18:0/880380009 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fc4000524e0 con 0x7fc4081430c0 2024-03-25T19:45:41.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.934+0000 7fc3ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.934+0000 7fc3ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:41.935 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:45:41.935 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:41.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 10 librados: watch_flush enter 2024-03-25T19:45:41.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 10 librados: watch_flush exit 2024-03-25T19:45:41.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T19:45:41.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 10 client.4412.objecter successfully canceled tick 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 >> v1:172.21.15.18:6800/1520000573 conn(0x7fc3f406e080 legacy=0x7fc3f4070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 10 monclient: shutdown 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 >> v1:172.21.15.18:6790/0 conn(0x7fc4081430c0 legacy=0x7fc4081333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 shutdown_connections 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 >> 172.21.15.18:0/880380009 conn(0x7fc4080b7ea0 msgr2=0x7fc408074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 shutdown_connections 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 1 -- 172.21.15.18:0/880380009 wait complete. 2024-03-25T19:45:41.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:41.937+0000 7fc40d30d640 1 librados: shutdown 2024-03-25T19:45:41.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:47.680 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:15.210 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:47.686 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:15.217 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:47.693 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:15.189 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:47.700 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:15.197 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:47.732 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:15.263 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:47.740 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:15.236 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:47.747 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:15.278 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:47.760 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:15.290 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:47.781 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:15.312 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:48.754 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.861+0000 7f9bf4e84640 10 monclient: get_monmap_and_config 2024-03-25T19:45:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.861+0000 7f9bf4e84640 10 monclient: build_initial_monmap 2024-03-25T19:45:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.861+0000 7f9bf4e84640 10 monclient: monmap: 2024-03-25T19:45:48.861 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:45:48.862607+0000 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:45:48.862607+0000 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:45:48.862 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:48.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.862+0000 7f9bf4e84640 1 Processor -- start 2024-03-25T19:45:48.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.862+0000 7f9bf4e84640 1 -- start start 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.862+0000 7f9bf4e84640 10 monclient: init 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf00a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf0142520 con 0x7f9bf01430c0 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00b28a0 con 0x7f9bf00a98f0 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00b2a30 con 0x7f9bf013f710 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:48.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.863+0000 7f9bf4e84640 10 monclient(hunting): authenticate will time out at 5175.990606s 2024-03-25T19:45:48.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.864+0000 7f9bee59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9bf00a98f0 0x7f9bf00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:60926/0 (socket says 172.21.15.18:60926) 2024-03-25T19:45:48.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.864+0000 7f9bee59c640 1 -- 172.21.15.18:0/2832727453 learned_addr learned my addr 172.21.15.18:0/2832727453 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.921+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060808742 0 0) 0x7f9bf00b2a30 con 0x7f9bf013f710 2024-03-25T19:45:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.921+0000 7f9bed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.921+0000 7f9bed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.921+0000 7f9bed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.921+0000 7f9bed59a640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.921+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8003520 con 0x7f9bf013f710 2024-03-25T19:45:48.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.934+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223196460 0 0) 0x7f9bf0142520 con 0x7f9bf01430c0 2024-03-25T19:45:48.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.934+0000 7f9bed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.934+0000 7f9bed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.934+0000 7f9bed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.934+0000 7f9bed59a640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:45:48.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.934+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf00b2a30 con 0x7f9bf01430c0 2024-03-25T19:45:48.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393353744 0 0) 0x7f9bd8003520 con 0x7f9bf013f710 2024-03-25T19:45:48.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 >> v1:172.21.15.53:6789/0 conn(0x7f9bf00a98f0 legacy=0x7f9bf00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 >> v1:172.21.15.18:6789/0 conn(0x7f9bf01430c0 legacy=0x7f9bf014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 10 monclient: _finish_hunting 0 2024-03-25T19:45:48.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 1 monclient: found mon.noname-b 2024-03-25T19:45:48.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:48.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:45:48.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00b44c0 con 0x7f9bf013f710 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 10 monclient: _finish_auth 0 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 10 monclient: _check_auth_tickets 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9be4001690 con 0x7f9bf013f710 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bf4e84640 5 monclient: authenticate success, global_id 4415 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bf4e84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bed59a640 10 monclient: dump: 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:48.948 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:48.949 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.949 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:48.949 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:48.949 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:48.949 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:48.949 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:48.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.947+0000 7f9bf4e84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:48.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.952+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9be40017f0 con 0x7f9bf013f710 2024-03-25T19:45:48.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.952+0000 7f9bed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:48.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.952+0000 7f9bed59a640 1 -- 172.21.15.18:0/2832727453 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9be4001c00 con 0x7f9bf013f710 2024-03-25T19:45:48.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.952+0000 7f9bf4e84640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:48.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.952+0000 7f9bf4e84640 4 set_mon_vals no callback set 2024-03-25T19:45:48.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.952+0000 7f9bed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.952+0000 7f9bed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:48.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.952+0000 7f9bed59a640 10 monclient: dump: 2024-03-25T19:45:48.953 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:48.953 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:48.953 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:48.954 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:48.954 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.954 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:48.954 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:48.954 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:48.954 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:48.954 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:48.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.953+0000 7f9bf4e84640 10 monclient: shutdown 2024-03-25T19:45:48.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.953+0000 7f9bf4e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.953+0000 7f9bf4e84640 1 -- 172.21.15.18:0/2832727453 >> v1:172.21.15.18:6790/0 conn(0x7f9bf013f710 legacy=0x7f9bf01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.953+0000 7f9bf4e84640 1 -- 172.21.15.18:0/2832727453 shutdown_connections 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.953+0000 7f9bf4e84640 1 -- 172.21.15.18:0/2832727453 >> 172.21.15.18:0/2832727453 conn(0x7f9bf00b7ea0 msgr2=0x7f9bf00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.953+0000 7f9bf4e84640 1 -- 172.21.15.18:0/2832727453 shutdown_connections 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.953+0000 7f9bf4e84640 1 -- 172.21.15.18:0/2832727453 wait complete. 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.954+0000 7f9bf4e84640 10 monclient: build_initial_monmap 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.954+0000 7f9bf4e84640 10 monclient: monmap: 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:45:48.958 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 librados: starting msgr at 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 librados: starting objecter 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 Processor -- start 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 -- start start 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 librados: setting wanted keys 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 librados: calling monclient init 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 10 monclient: init 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:48.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:48.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00aba00 con 0x7f9bf00a98f0 2024-03-25T19:45:48.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf01d7490 con 0x7f9bf013f710 2024-03-25T19:45:48.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf01d8360 con 0x7f9bf01430c0 2024-03-25T19:45:48.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:48.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.955+0000 7f9bf4e84640 10 monclient(hunting): authenticate will time out at 5176.082663s 2024-03-25T19:45:48.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.956+0000 7f9bee59c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f9bf00a98f0 0x7f9bf01386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:34064/0 (socket says 172.21.15.18:34064) 2024-03-25T19:45:48.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.956+0000 7f9bee59c640 1 -- 172.21.15.18:0/1080446915 learned_addr learned my addr 172.21.15.18:0/1080446915 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:48.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.959+0000 7f9bdeffd640 10 client.?.objecter ms_handle_connect 0x7f9bf00a98f0 2024-03-25T19:45:48.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.959+0000 7f9bdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:48.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.962+0000 7f9bdeffd640 10 client.?.objecter ms_handle_connect 0x7f9bf01430c0 2024-03-25T19:45:48.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.962+0000 7f9bdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:48.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.974+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419880542 0 0) 0x7f9bf00aba00 con 0x7f9bf00a98f0 2024-03-25T19:45:48.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.974+0000 7f9bdeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.974+0000 7f9bdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.974+0000 7f9bdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.974+0000 7f9bdeffd640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:45:48.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.974+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd0004eb0 con 0x7f9bf00a98f0 2024-03-25T19:45:48.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.975+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595318888 0 0) 0x7f9bf01d8360 con 0x7f9bf01430c0 2024-03-25T19:45:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.975+0000 7f9bdeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.975+0000 7f9bdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.975+0000 7f9bdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.975+0000 7f9bdeffd640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:45:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.975+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf00aba00 con 0x7f9bf01430c0 2024-03-25T19:45:48.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.989+0000 7f9bdeffd640 10 client.?.objecter ms_handle_connect 0x7f9bf013f710 2024-03-25T19:45:48.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:48.989+0000 7f9bdeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:49.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.000+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136257398 0 0) 0x7f9bf01d7490 con 0x7f9bf013f710 2024-03-25T19:45:49.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.000+0000 7f9bdeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:49.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.000+0000 7f9bdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:49.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.000+0000 7f9bdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:49.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.000+0000 7f9bdeffd640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:45:49.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.000+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf01d8360 con 0x7f9bf013f710 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.005+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 955954219 0 0) 0x7f9bd0004eb0 con 0x7f9bf00a98f0 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.005+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf01d7490 con 0x7f9bf00a98f0 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.005+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9be00018b0 con 0x7f9bf00a98f0 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.005+0000 7f9bdeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.005+0000 7f9bdeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.005+0000 7f9bdeffd640 10 monclient(hunting): dump: 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:49.005 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:49.006 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:49.006 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:49.006 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:49.006 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:49.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.005+0000 7f9bdeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3313205188 0 0) 0x7f9bf01d7490 con 0x7f9bf00a98f0 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 >> v1:172.21.15.18:6790/0 conn(0x7f9bf01430c0 legacy=0x7f9bf0133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 >> v1:172.21.15.53:6789/0 conn(0x7f9bf013f710 legacy=0x7f9bf01d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 1 monclient: found mon.a 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00abc30 con 0x7f9bf00a98f0 2024-03-25T19:45:49.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 5 monclient: authenticate success, global_id 4449 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 10 monclient: _renew_subs 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf0137940 con 0x7f9bf00a98f0 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 10 monclient: _renew_subs 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf0135220 con 0x7f9bf00a98f0 2024-03-25T19:45:49.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.019+0000 7f9bf4e84640 1 librados: init done 2024-03-25T19:45:49.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.020+0000 7f9bf4e84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:49.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.020+0000 7f9bf4e84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:49.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.020+0000 7f9bf4e84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:45:49.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.020+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf00627b0 con 0x7f9bf00a98f0 2024-03-25T19:45:49.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.325+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9be00047e0 con 0x7f9bf00a98f0 2024-03-25T19:45:49.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.325+0000 7f9bdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.325+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9be0004c30 con 0x7f9bf00a98f0 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.325+0000 7f9bdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.325+0000 7f9bdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.325+0000 7f9bdeffd640 10 monclient: dump: 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:49.326 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:49.327 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:49.327 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:49.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.325+0000 7f9becd99640 4 set_mon_vals no callback set 2024-03-25T19:45:49.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.326+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f9be0004e50 con 0x7f9bf00a98f0 2024-03-25T19:45:49.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.327+0000 7f9bedd9b640 10 monclient: get_auth_request con 0x7f9bd006ddf0 auth_method 0 2024-03-25T19:45:49.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.327+0000 7f9bdeffd640 10 client.4449.objecter ms_handle_connect 0x7f9bd006ddf0 2024-03-25T19:45:49.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.334+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f9be0054830 con 0x7f9bf00a98f0 2024-03-25T19:45:49.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.334+0000 7f9bdeffd640 10 client.4449.objecter ms_dispatch 0x7f9bf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:49.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.334+0000 7f9bdeffd640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:49.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.334+0000 7f9bdeffd640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:49.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.334+0000 7f9bdeffd640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-25T19:45:49.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.334+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9be0050340 con 0x7f9bf00a98f0 2024-03-25T19:45:49.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.334+0000 7f9bdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:49.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.334+0000 7f9bdeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:49.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.583+0000 7f9bf4e84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.583+0000 7f9bf4e84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.583+0000 7f9bf4e84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:45:49.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.583+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bf01d7830 con 0x7f9bf00a98f0 2024-03-25T19:45:49.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.607+0000 7f9bdeffd640 1 -- 172.21.15.18:0/1080446915 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f9be0052270 con 0x7f9bf00a98f0 2024-03-25T19:45:49.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.607+0000 7f9bdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.607+0000 7f9bdeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:49.607 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:45:49.607 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:49.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 10 librados: watch_flush enter 2024-03-25T19:45:49.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 10 librados: watch_flush exit 2024-03-25T19:45:49.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-25T19:45:49.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 10 client.4449.objecter successfully canceled tick 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 >> v1:172.21.15.18:6800/1520000573 conn(0x7f9bd006ddf0 legacy=0x7f9bd00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 10 monclient: shutdown 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 >> v1:172.21.15.18:6789/0 conn(0x7f9bf00a98f0 legacy=0x7f9bf01386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 shutdown_connections 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 >> 172.21.15.18:0/1080446915 conn(0x7f9bf00b7ea0 msgr2=0x7f9bf0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 shutdown_connections 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 1 -- 172.21.15.18:0/1080446915 wait complete. 2024-03-25T19:45:49.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:49.609+0000 7f9bf4e84640 1 librados: shutdown 2024-03-25T19:45:49.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:56.427 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:56.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.543+0000 7fdf8f8e4640 10 monclient: get_monmap_and_config 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.543+0000 7fdf8f8e4640 10 monclient: build_initial_monmap 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.543+0000 7fdf8f8e4640 10 monclient: monmap: 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:45:56.543964+0000 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:45:56.543964+0000 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:45:56.543 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:56.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.543+0000 7fdf8f8e4640 1 Processor -- start 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 1 -- start start 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 10 monclient: init 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 10 monclient(hunting): picked mon.noname-b con 0x7fdf880a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 10 monclient(hunting): picked mon.noname-c con 0x7fdf8813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 10 monclient(hunting): picked mon.noname-a con 0x7fdf881430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf88142520 con 0x7fdf881430c0 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf880b28a0 con 0x7fdf8813f710 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf880b2a30 con 0x7fdf880a98f0 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.544+0000 7fdf8f8e4640 10 monclient(hunting): authenticate will time out at 5183.671699s 2024-03-25T19:45:56.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.549+0000 7fdf8d680640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fdf880a98f0 0x7fdf880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55398/0 (socket says 172.21.15.18:55398) 2024-03-25T19:45:56.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.549+0000 7fdf8d680640 1 -- 172.21.15.18:0/1573641781 learned_addr learned my addr 172.21.15.18:0/1573641781 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:56.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.567+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372088809 0 0) 0x7fdf880b2a30 con 0x7fdf880a98f0 2024-03-25T19:45:56.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.567+0000 7fdf7ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.567+0000 7fdf7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.567+0000 7fdf7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.567+0000 7fdf7ffff640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:45:56.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.567+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf74003520 con 0x7fdf880a98f0 2024-03-25T19:45:56.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.574+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724909049 0 0) 0x7fdf88142520 con 0x7fdf881430c0 2024-03-25T19:45:56.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.574+0000 7fdf7ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.574+0000 7fdf7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.574+0000 7fdf7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.574+0000 7fdf7ffff640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:45:56.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.574+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf880b2a30 con 0x7fdf881430c0 2024-03-25T19:45:56.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.599+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134132311 0 0) 0x7fdf880b28a0 con 0x7fdf8813f710 2024-03-25T19:45:56.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.599+0000 7fdf7ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.599+0000 7fdf7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.599+0000 7fdf7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.599+0000 7fdf7ffff640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:45:56.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.599+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf88142520 con 0x7fdf8813f710 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137076257 0 0) 0x7fdf74003520 con 0x7fdf880a98f0 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 >> v1:172.21.15.53:6789/0 conn(0x7fdf8813f710 legacy=0x7fdf881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 >> v1:172.21.15.18:6789/0 conn(0x7fdf881430c0 legacy=0x7fdf8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 1 monclient: found mon.noname-b 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf880b44c0 con 0x7fdf880a98f0 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 10 monclient: _finish_auth 0 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fdf84001690 con 0x7fdf880a98f0 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf7ffff640 10 monclient: dump: 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:56.605 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:56.606 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:56.606 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:56.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf8f8e4640 5 monclient: authenticate success, global_id 4421 2024-03-25T19:45:56.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.604+0000 7fdf8f8e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:56.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf840017f0 con 0x7fdf880a98f0 2024-03-25T19:45:56.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf7ffff640 1 -- 172.21.15.18:0/1573641781 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fdf84001c00 con 0x7fdf880a98f0 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf8f8e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf8f8e4640 4 set_mon_vals no callback set 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf7ffff640 10 monclient: dump: 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:56.624 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf8f8e4640 10 monclient: shutdown 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf8f8e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/1573641781 >> v1:172.21.15.18:6790/0 conn(0x7fdf880a98f0 legacy=0x7fdf880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/1573641781 shutdown_connections 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.621+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/1573641781 >> 172.21.15.18:0/1573641781 conn(0x7fdf880b7ea0 msgr2=0x7fdf880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.622+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/1573641781 shutdown_connections 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.622+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/1573641781 wait complete. 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.622+0000 7fdf8f8e4640 10 monclient: build_initial_monmap 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.622+0000 7fdf8f8e4640 10 monclient: monmap: 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:45:56.625 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 librados: starting msgr at 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 librados: starting objecter 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 Processor -- start 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 -- start start 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 librados: setting wanted keys 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 librados: calling monclient init 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 10 monclient: init 2024-03-25T19:45:56.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 10 monclient(hunting): picked mon.noname-a con 0x7fdf880a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 10 monclient(hunting): picked mon.noname-c con 0x7fdf8813f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 10 monclient(hunting): picked mon.noname-b con 0x7fdf881430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf88138b20 con 0x7fdf880a98f0 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf88137a80 con 0x7fdf881430c0 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf88139880 con 0x7fdf8813f710 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:56.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.623+0000 7fdf8f8e4640 10 monclient(hunting): authenticate will time out at 5183.750549s 2024-03-25T19:45:56.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.629+0000 7fdf8de81640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdf881430c0 0x7fdf881e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:33836/0 (socket says 172.21.15.18:33836) 2024-03-25T19:45:56.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.629+0000 7fdf8de81640 1 -- 172.21.15.18:0/2997925181 learned_addr learned my addr 172.21.15.18:0/2997925181 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:45:56.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.633+0000 7fdf7dffb640 10 client.?.objecter ms_handle_connect 0x7fdf881430c0 2024-03-25T19:45:56.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.633+0000 7fdf7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.634+0000 7fdf7dffb640 10 client.?.objecter ms_handle_connect 0x7fdf880a98f0 2024-03-25T19:45:56.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.634+0000 7fdf7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.637+0000 7fdf7dffb640 10 client.?.objecter ms_handle_connect 0x7fdf8813f710 2024-03-25T19:45:56.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.637+0000 7fdf7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.640+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194761060 0 0) 0x7fdf88137a80 con 0x7fdf881430c0 2024-03-25T19:45:56.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.640+0000 7fdf7dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.640+0000 7fdf7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.640+0000 7fdf7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.640+0000 7fdf7dffb640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:45:56.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.640+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6c004eb0 con 0x7fdf881430c0 2024-03-25T19:45:56.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.651+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778228348 0 0) 0x7fdf88138b20 con 0x7fdf880a98f0 2024-03-25T19:45:56.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.651+0000 7fdf7dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.651+0000 7fdf7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.651+0000 7fdf7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.651+0000 7fdf7dffb640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:45:56.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.651+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf88137a80 con 0x7fdf880a98f0 2024-03-25T19:45:56.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.652+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857209579 0 0) 0x7fdf88139880 con 0x7fdf8813f710 2024-03-25T19:45:56.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.652+0000 7fdf7dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.652+0000 7fdf7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.652+0000 7fdf7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.652+0000 7fdf7dffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:45:56.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.652+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf88138b20 con 0x7fdf8813f710 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.661+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460114606 0 0) 0x7fdf6c004eb0 con 0x7fdf881430c0 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.661+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf88139880 con 0x7fdf881430c0 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.661+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fdf80001c90 con 0x7fdf881430c0 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.661+0000 7fdf7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.661+0000 7fdf7dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.661+0000 7fdf7dffb640 10 monclient(hunting): dump: 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:56.661 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.662 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:56.662 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:56.662 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:56.662 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:56.662 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:56.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.661+0000 7fdf7dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:56.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 480240050 0 0) 0x7fdf88139880 con 0x7fdf881430c0 2024-03-25T19:45:56.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 >> v1:172.21.15.18:6790/0 conn(0x7fdf8813f710 legacy=0x7fdf88139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 >> v1:172.21.15.18:6789/0 conn(0x7fdf880a98f0 legacy=0x7fdf88137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 1 monclient: found mon.b 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88139a10 con 0x7fdf881430c0 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 5 monclient: authenticate success, global_id 4423 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 10 monclient: _renew_subs 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf881dab00 con 0x7fdf881430c0 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 10 monclient: _renew_subs 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:56.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8813b4f0 con 0x7fdf881430c0 2024-03-25T19:45:56.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.678+0000 7fdf8f8e4640 1 librados: init done 2024-03-25T19:45:56.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.679+0000 7fdf8f8e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:56.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.679+0000 7fdf8f8e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:56.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.679+0000 7fdf8f8e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:56.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:56.679+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf880627b0 con 0x7fdf881430c0 2024-03-25T19:45:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.002+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf80003970 con 0x7fdf881430c0 2024-03-25T19:45:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.002+0000 7fdf7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.002+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fdf80003d80 con 0x7fdf881430c0 2024-03-25T19:45:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.002+0000 7fdf7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.002+0000 7fdf7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.002+0000 7fdf7dffb640 10 monclient: dump: 2024-03-25T19:45:57.002 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:45:57.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.002+0000 7fdf7f7fe640 4 set_mon_vals no callback set 2024-03-25T19:45:57.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.003+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fdf80003fa0 con 0x7fdf881430c0 2024-03-25T19:45:57.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.004+0000 7fdf8d680640 10 monclient: get_auth_request con 0x7fdf6c06ddf0 auth_method 0 2024-03-25T19:45:57.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.004+0000 7fdf7dffb640 10 client.4423.objecter ms_handle_connect 0x7fdf6c06ddf0 2024-03-25T19:45:57.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.011+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fdf80054ae0 con 0x7fdf881430c0 2024-03-25T19:45:57.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.011+0000 7fdf7dffb640 10 client.4423.objecter ms_dispatch 0x7fdf88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:57.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.011+0000 7fdf7dffb640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:57.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.011+0000 7fdf7dffb640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:57.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.011+0000 7fdf7dffb640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T19:45:57.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.011+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf800505f0 con 0x7fdf881430c0 2024-03-25T19:45:57.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.011+0000 7fdf7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:57.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.011+0000 7fdf7dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:57.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.246+0000 7fdf8f8e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:57.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.247+0000 7fdf8f8e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:57.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.247+0000 7fdf8f8e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:45:57.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.247+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf880b3e80 con 0x7fdf881430c0 2024-03-25T19:45:57.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.274+0000 7fdf7dffb640 1 -- 172.21.15.18:0/2997925181 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fdf80052520 con 0x7fdf881430c0 2024-03-25T19:45:57.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.274+0000 7fdf7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:57.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.274+0000 7fdf7dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:57.274 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:45:57.274 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:57.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.276+0000 7fdf8f8e4640 10 librados: watch_flush enter 2024-03-25T19:45:57.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 10 librados: watch_flush exit 2024-03-25T19:45:57.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T19:45:57.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 10 client.4423.objecter successfully canceled tick 2024-03-25T19:45:57.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 >> v1:172.21.15.18:6800/1520000573 conn(0x7fdf6c06ddf0 legacy=0x7fdf6c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 10 monclient: shutdown 2024-03-25T19:45:57.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:57.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 >> v1:172.21.15.53:6789/0 conn(0x7fdf881430c0 legacy=0x7fdf881e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 shutdown_connections 2024-03-25T19:45:57.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 >> 172.21.15.18:0/2997925181 conn(0x7fdf880b7ea0 msgr2=0x7fdf88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:57.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 shutdown_connections 2024-03-25T19:45:57.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 1 -- 172.21.15.18:0/2997925181 wait complete. 2024-03-25T19:45:57.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:45:57.277+0000 7fdf8f8e4640 1 librados: shutdown 2024-03-25T19:45:57.289 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:04.095 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:04.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.204+0000 7f84ad827640 10 monclient: get_monmap_and_config 2024-03-25T19:46:04.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.204+0000 7f84ad827640 10 monclient: build_initial_monmap 2024-03-25T19:46:04.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.204+0000 7f84ad827640 10 monclient: monmap: 2024-03-25T19:46:04.204 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:04.204 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:04.204 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:46:04.205199+0000 2024-03-25T19:46:04.204 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:46:04.205199+0000 2024-03-25T19:46:04.205 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:04.205 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:04.205 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:04.205 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:46:04.205 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:46:04.205 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:04.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.204+0000 7f84ad827640 1 Processor -- start 2024-03-25T19:46:04.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.205+0000 7f84ad827640 1 -- start start 2024-03-25T19:46:04.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.205+0000 7f84ad827640 10 monclient: init 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.205+0000 7f84ad827640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.205+0000 7f84ad827640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84ad827640 10 monclient(hunting): picked mon.noname-b con 0x7f84a80b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84ad827640 10 monclient(hunting): picked mon.noname-a con 0x7f84a80b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84ad827640 10 monclient(hunting): picked mon.noname-c con 0x7f84a814be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84ad827640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a80af940 con 0x7f84a80b1730 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84ad827640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a80b0a70 con 0x7f84a814be90 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84ad827640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a80b0d50 con 0x7f84a80b4d60 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84ad827640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84ad827640 10 monclient(hunting): authenticate will time out at 5191.332839s 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84a77fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f84a814be90 0x7f84a814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:38350/0 (socket says 172.21.15.18:38350) 2024-03-25T19:46:04.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.206+0000 7f84a77fe640 1 -- 172.21.15.18:0/740812749 learned_addr learned my addr 172.21.15.18:0/740812749 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:04.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.264+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046983823 0 0) 0x7f84a80af940 con 0x7f84a80b1730 2024-03-25T19:46:04.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.264+0000 7f84a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.264+0000 7f84a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.264+0000 7f84a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.264+0000 7f84a67fc640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:46:04.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.264+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8494003520 con 0x7f84a80b1730 2024-03-25T19:46:04.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.266+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544919076 0 0) 0x7f84a80b0d50 con 0x7f84a80b4d60 2024-03-25T19:46:04.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.266+0000 7f84a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.266+0000 7f84a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.266+0000 7f84a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.266+0000 7f84a67fc640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:46:04.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.266+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a80af940 con 0x7f84a80b4d60 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1585307009 0 0) 0x7f8494003520 con 0x7f84a80b1730 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 >> v1:172.21.15.18:6790/0 conn(0x7f84a80b4d60 legacy=0x7f84a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 >> v1:172.21.15.53:6789/0 conn(0x7f84a814be90 legacy=0x7f84a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 1 monclient: found mon.noname-a 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a80b1030 con 0x7f84a80b1730 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:04.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8490001690 con 0x7f84a80b1730 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84a67fc640 10 monclient: dump: 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:04.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84ad827640 5 monclient: authenticate success, global_id 4458 2024-03-25T19:46:04.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.281+0000 7f84ad827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:04.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.304+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84900017f0 con 0x7f84a80b1730 2024-03-25T19:46:04.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.304+0000 7f84a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:04.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.304+0000 7f84a67fc640 1 -- 172.21.15.18:0/740812749 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8490001c00 con 0x7f84a80b1730 2024-03-25T19:46:04.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.304+0000 7f84ad827640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:04.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.304+0000 7f84ad827640 4 set_mon_vals no callback set 2024-03-25T19:46:04.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.304+0000 7f84a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.304+0000 7f84a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.304+0000 7f84a67fc640 10 monclient: dump: 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:04.305 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.305+0000 7f84ad827640 10 monclient: shutdown 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.305+0000 7f84ad827640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.305+0000 7f84ad827640 1 -- 172.21.15.18:0/740812749 >> v1:172.21.15.18:6789/0 conn(0x7f84a80b1730 legacy=0x7f84a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.305+0000 7f84ad827640 1 -- 172.21.15.18:0/740812749 shutdown_connections 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.305+0000 7f84ad827640 1 -- 172.21.15.18:0/740812749 >> 172.21.15.18:0/740812749 conn(0x7f84a80617a0 msgr2=0x7f84a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.305+0000 7f84ad827640 1 -- 172.21.15.18:0/740812749 shutdown_connections 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.305+0000 7f84ad827640 1 -- 172.21.15.18:0/740812749 wait complete. 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 10 monclient: build_initial_monmap 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 10 monclient: monmap: 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:46:04.308 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 1 librados: starting msgr at 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 1 librados: starting objecter 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 1 Processor -- start 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 1 -- start start 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 1 librados: setting wanted keys 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 1 librados: calling monclient init 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.306+0000 7f84ad827640 10 monclient: init 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:04.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:04.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 10 monclient(hunting): picked mon.noname-a con 0x7f84a80b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:04.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 10 monclient(hunting): picked mon.noname-c con 0x7f84a80b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:04.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 10 monclient(hunting): picked mon.noname-b con 0x7f84a814be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:04.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a8137fa0 con 0x7f84a80b1730 2024-03-25T19:46:04.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a81361c0 con 0x7f84a814be90 2024-03-25T19:46:04.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a8135ca0 con 0x7f84a80b4d60 2024-03-25T19:46:04.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:04.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.307+0000 7f84ad827640 10 monclient(hunting): authenticate will time out at 5191.434692s 2024-03-25T19:46:04.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.311+0000 7f849e7fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f84a80b4d60 0x7f84a81dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:60160/0 (socket says 172.21.15.18:60160) 2024-03-25T19:46:04.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.311+0000 7f849e7fc640 1 -- 172.21.15.18:0/726450897 learned_addr learned my addr 172.21.15.18:0/726450897 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.315+0000 7f84a4ff9640 10 client.?.objecter ms_handle_connect 0x7f84a80b4d60 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.315+0000 7f84a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:04.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.317+0000 7f84a4ff9640 10 client.?.objecter ms_handle_connect 0x7f84a814be90 2024-03-25T19:46:04.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.317+0000 7f84a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:04.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.329+0000 7f84a4ff9640 10 client.?.objecter ms_handle_connect 0x7f84a80b1730 2024-03-25T19:46:04.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.329+0000 7f84a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:04.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.333+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067453793 0 0) 0x7f84a81361c0 con 0x7f84a814be90 2024-03-25T19:46:04.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.333+0000 7f84a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.333+0000 7f84a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.333+0000 7f84a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.333+0000 7f84a4ff9640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:46:04.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.333+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8488004eb0 con 0x7f84a814be90 2024-03-25T19:46:04.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.334+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530003856 0 0) 0x7f84a8135ca0 con 0x7f84a80b4d60 2024-03-25T19:46:04.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.334+0000 7f84a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.334+0000 7f84a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.334+0000 7f84a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.334+0000 7f84a4ff9640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:46:04.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.334+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a81361c0 con 0x7f84a80b4d60 2024-03-25T19:46:04.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.344+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55487371 0 0) 0x7f84a8137fa0 con 0x7f84a80b1730 2024-03-25T19:46:04.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.344+0000 7f84a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.344+0000 7f84a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.344+0000 7f84a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.344+0000 7f84a4ff9640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:46:04.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.344+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a8135ca0 con 0x7f84a80b1730 2024-03-25T19:46:04.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.350+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1804137088 0 0) 0x7f84a81361c0 con 0x7f84a80b4d60 2024-03-25T19:46:04.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.350+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84a8137fa0 con 0x7f84a80b4d60 2024-03-25T19:46:04.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.350+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8490001690 con 0x7f84a80b4d60 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.350+0000 7f84a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.350+0000 7f84a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.350+0000 7f84a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:04.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.350+0000 7f84a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:04.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.354+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1802709183 0 0) 0x7f84a8135ca0 con 0x7f84a80b1730 2024-03-25T19:46:04.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.354+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84a81361c0 con 0x7f84a80b1730 2024-03-25T19:46:04.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.354+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8498001930 con 0x7f84a80b1730 2024-03-25T19:46:04.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.354+0000 7f84a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.354+0000 7f84a4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.354+0000 7f84a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:04.355 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.361+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3344927924 0 0) 0x7f84a81361c0 con 0x7f84a80b1730 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.361+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 >> v1:172.21.15.18:6790/0 conn(0x7f84a80b4d60 legacy=0x7f84a81dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.361+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 >> v1:172.21.15.53:6789/0 conn(0x7f84a814be90 legacy=0x7f84a81e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84a4ff9640 1 monclient: found mon.a 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a8138350 con 0x7f84a80b1730 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84a4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:04.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 5 monclient: authenticate success, global_id 4461 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 10 monclient: _renew_subs 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a8136410 con 0x7f84a80b1730 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 10 monclient: _renew_subs 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a8130770 con 0x7f84a80b1730 2024-03-25T19:46:04.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.362+0000 7f84ad827640 1 librados: init done 2024-03-25T19:46:04.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.363+0000 7f84ad827640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:04.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.363+0000 7f84ad827640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:04.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.363+0000 7f84ad827640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:04.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.363+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a80627b0 con 0x7f84a80b1730 2024-03-25T19:46:04.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.694+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8498003610 con 0x7f84a80b1730 2024-03-25T19:46:04.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.694+0000 7f84a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:04.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.694+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8498001d80 con 0x7f84a80b1730 2024-03-25T19:46:04.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.694+0000 7f84a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.694+0000 7f84a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.694+0000 7f84a4ff9640 10 monclient: dump: 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:04.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.694+0000 7f84a5ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:04.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.695+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f8498019810 con 0x7f84a80b1730 2024-03-25T19:46:04.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.696+0000 7f849e7fc640 10 monclient: get_auth_request con 0x7f848806df80 auth_method 0 2024-03-25T19:46:04.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.696+0000 7f84a4ff9640 10 client.4461.objecter ms_handle_connect 0x7f848806df80 2024-03-25T19:46:04.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.702+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f84980534e0 con 0x7f84a80b1730 2024-03-25T19:46:04.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.702+0000 7f84a4ff9640 10 client.4461.objecter ms_dispatch 0x7f84a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:04.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.702+0000 7f84a4ff9640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.702+0000 7f84a4ff9640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.702+0000 7f84a4ff9640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.702+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f849804ee20 con 0x7f84a80b1730 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.702+0000 7f84a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.702+0000 7f84a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:04.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.971+0000 7f84ad827640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:04.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.971+0000 7f84ad827640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:04.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.971+0000 7f84ad827640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:04.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:04.971+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84a81365a0 con 0x7f84a80b1730 2024-03-25T19:46:05.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.012+0000 7f84a4ff9640 1 -- 172.21.15.18:0/726450897 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f8498050f20 con 0x7f84a80b1730 2024-03-25T19:46:05.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.012+0000 7f84a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:05.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.012+0000 7f84a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:05.012 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:46:05.012 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:05.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 10 librados: watch_flush enter 2024-03-25T19:46:05.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 10 librados: watch_flush exit 2024-03-25T19:46:05.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T19:46:05.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 10 client.4461.objecter successfully canceled tick 2024-03-25T19:46:05.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 >> v1:172.21.15.18:6800/1520000573 conn(0x7f848806df80 legacy=0x7f8488070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 10 monclient: shutdown 2024-03-25T19:46:05.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:05.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 >> v1:172.21.15.18:6789/0 conn(0x7f84a80b1730 legacy=0x7f84a812fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:05.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 shutdown_connections 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 >> 172.21.15.18:0/726450897 conn(0x7f84a80617a0 msgr2=0x7f84a80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 shutdown_connections 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 1 -- 172.21.15.18:0/726450897 wait complete. 2024-03-25T19:46:05.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:05.014+0000 7f84ad827640 1 librados: shutdown 2024-03-25T19:46:05.027 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:07.689 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:35.219 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.741 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:35.272 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.790 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:35.321 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.796 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:35.292 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.807 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:35.303 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.822 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:35.352 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.829 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:35.325 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.831 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:35.361 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:07.852 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:35.383 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:11.835 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.944+0000 7f6560635640 10 monclient: get_monmap_and_config 2024-03-25T19:46:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.944+0000 7f6560635640 10 monclient: build_initial_monmap 2024-03-25T19:46:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.944+0000 7f6560635640 10 monclient: monmap: 2024-03-25T19:46:11.945 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:11.946 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:11.946 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:46:11.945301+0000 2024-03-25T19:46:11.946 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:46:11.945301+0000 2024-03-25T19:46:11.946 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:11.946 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:11.946 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:11.947 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:46:11.947 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:46:11.947 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:11.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.945+0000 7f6560635640 1 Processor -- start 2024-03-25T19:46:11.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.945+0000 7f6560635640 1 -- start start 2024-03-25T19:46:11.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.945+0000 7f6560635640 10 monclient: init 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.945+0000 7f6560635640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.945+0000 7f6560635640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.945+0000 7f6560635640 10 monclient(hunting): picked mon.noname-b con 0x7f65580a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.946+0000 7f6560635640 10 monclient(hunting): picked mon.noname-c con 0x7f655813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.946+0000 7f6560635640 10 monclient(hunting): picked mon.noname-a con 0x7f65581430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.946+0000 7f6560635640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6558142520 con 0x7f65581430c0 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.946+0000 7f6560635640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65580b28a0 con 0x7f655813f710 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.946+0000 7f6560635640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65580b2a30 con 0x7f65580a98f0 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.946+0000 7f6560635640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.946+0000 7f6560635640 10 monclient(hunting): authenticate will time out at 5199.072822s 2024-03-25T19:46:11.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.953+0000 7f655dbd0640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f655813f710 0x7f65581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:33070/0 (socket says 172.21.15.18:33070) 2024-03-25T19:46:11.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.953+0000 7f655dbd0640 1 -- 172.21.15.18:0/554192288 learned_addr learned my addr 172.21.15.18:0/554192288 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:11.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.963+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872900980 0 0) 0x7f65580b28a0 con 0x7f655813f710 2024-03-25T19:46:11.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.963+0000 7f655d3cf640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.963+0000 7f655d3cf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.963+0000 7f655d3cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.963+0000 7f655d3cf640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.963+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6544003520 con 0x7f655813f710 2024-03-25T19:46:11.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.976+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249848448 0 0) 0x7f65580b2a30 con 0x7f65580a98f0 2024-03-25T19:46:11.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.976+0000 7f655d3cf640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.976+0000 7f655d3cf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.976+0000 7f655d3cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.976+0000 7f655d3cf640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:46:11.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.976+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65580b28a0 con 0x7f65580a98f0 2024-03-25T19:46:11.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2320587042 0 0) 0x7f6544003520 con 0x7f655813f710 2024-03-25T19:46:11.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 >> v1:172.21.15.18:6790/0 conn(0x7f65580a98f0 legacy=0x7f65580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 >> v1:172.21.15.18:6789/0 conn(0x7f65581430c0 legacy=0x7f655814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 10 monclient: _finish_hunting 0 2024-03-25T19:46:11.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 1 monclient: found mon.noname-c 2024-03-25T19:46:11.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:11.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:46:11.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65580b44c0 con 0x7f655813f710 2024-03-25T19:46:11.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 10 monclient: _finish_auth 0 2024-03-25T19:46:11.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 10 monclient: _check_auth_tickets 2024-03-25T19:46:11.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:11.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6548001690 con 0x7f655813f710 2024-03-25T19:46:11.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:11.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f655d3cf640 10 monclient: dump: 2024-03-25T19:46:11.982 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:11.982 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:11.982 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:11.982 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:11.983 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.983 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:11.983 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:11.983 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:11.983 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:11.983 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:11.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f6560635640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:46:11.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.978+0000 7f6560635640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:11.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.985+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65480017f0 con 0x7f655813f710 2024-03-25T19:46:11.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.985+0000 7f655d3cf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:11.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.985+0000 7f655d3cf640 1 -- 172.21.15.18:0/554192288 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6548001c00 con 0x7f655813f710 2024-03-25T19:46:11.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.985+0000 7f655d3cf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.985+0000 7f655d3cf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:11.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.985+0000 7f655d3cf640 10 monclient: dump: 2024-03-25T19:46:11.987 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:11.987 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:11.987 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:11.987 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:11.987 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.987 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:11.988 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:11.988 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:11.988 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:11.988 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:11.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.985+0000 7f6560635640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:11.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.985+0000 7f6560635640 4 set_mon_vals no callback set 2024-03-25T19:46:11.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.986+0000 7f6560635640 10 monclient: shutdown 2024-03-25T19:46:11.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.986+0000 7f6560635640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:11.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.986+0000 7f6560635640 1 -- 172.21.15.18:0/554192288 >> v1:172.21.15.53:6789/0 conn(0x7f655813f710 legacy=0x7f65581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.986+0000 7f6560635640 1 -- 172.21.15.18:0/554192288 shutdown_connections 2024-03-25T19:46:11.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.986+0000 7f6560635640 1 -- 172.21.15.18:0/554192288 >> 172.21.15.18:0/554192288 conn(0x7f65580b7ea0 msgr2=0x7f65580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.986+0000 7f6560635640 1 -- 172.21.15.18:0/554192288 shutdown_connections 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.986+0000 7f6560635640 1 -- 172.21.15.18:0/554192288 wait complete. 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.987+0000 7f6560635640 10 monclient: build_initial_monmap 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.987+0000 7f6560635640 10 monclient: monmap: 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:11.995 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:11.995 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:46:11.995 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:46:11.995 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:11.995 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:46:11.995 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:11.996 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:46:11.996 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:46:11.996 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:11.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 1 librados: starting msgr at 2024-03-25T19:46:11.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 1 librados: starting objecter 2024-03-25T19:46:11.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 1 Processor -- start 2024-03-25T19:46:11.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 1 -- start start 2024-03-25T19:46:11.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 1 librados: setting wanted keys 2024-03-25T19:46:11.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 1 librados: calling monclient init 2024-03-25T19:46:11.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 10 monclient: init 2024-03-25T19:46:11.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:11.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:11.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 10 monclient(hunting): picked mon.noname-b con 0x7f65580a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:11.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 10 monclient(hunting): picked mon.noname-c con 0x7f655813f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:11.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 10 monclient(hunting): picked mon.noname-a con 0x7f65581430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:11.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.988+0000 7f6560635640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65581d77a0 con 0x7f65581430c0 2024-03-25T19:46:11.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.989+0000 7f6560635640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6558138e40 con 0x7f65580a98f0 2024-03-25T19:46:11.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.989+0000 7f6560635640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65580aba30 con 0x7f655813f710 2024-03-25T19:46:11.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.989+0000 7f6560635640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:11.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.989+0000 7f6560635640 10 monclient(hunting): authenticate will time out at 5199.115767s 2024-03-25T19:46:12.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.994+0000 7f655dbd0640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f655813f710 0x7f65581d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:44888/0 (socket says 172.21.15.18:44888) 2024-03-25T19:46:12.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.994+0000 7f655dbd0640 1 -- 172.21.15.18:0/3814337750 learned_addr learned my addr 172.21.15.18:0/3814337750 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:12.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.995+0000 7f654effd640 10 client.?.objecter ms_handle_connect 0x7f655813f710 2024-03-25T19:46:12.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:11.995+0000 7f654effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:12.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.000+0000 7f654effd640 10 client.?.objecter ms_handle_connect 0x7f65580a98f0 2024-03-25T19:46:12.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.000+0000 7f654effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:12.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.009+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374447713 0 0) 0x7f6558138e40 con 0x7f65580a98f0 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.009+0000 7f654effd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.009+0000 7f654effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.009+0000 7f654effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.009+0000 7f654effd640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.009+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f653c004eb0 con 0x7f65580a98f0 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.010+0000 7f654effd640 10 client.?.objecter ms_handle_connect 0x7f65581430c0 2024-03-25T19:46:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.010+0000 7f654effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:12.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.016+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905935905 0 0) 0x7f65580aba30 con 0x7f655813f710 2024-03-25T19:46:12.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.016+0000 7f654effd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:12.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.016+0000 7f654effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:12.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.016+0000 7f654effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:12.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.016+0000 7f654effd640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:46:12.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.016+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6558138e40 con 0x7f655813f710 2024-03-25T19:46:12.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.025+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408227038 0 0) 0x7f65581d77a0 con 0x7f65581430c0 2024-03-25T19:46:12.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.025+0000 7f654effd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:12.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.025+0000 7f654effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:12.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.025+0000 7f654effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:12.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.025+0000 7f654effd640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:46:12.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.025+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65580aba30 con 0x7f65581430c0 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.033+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693659504 0 0) 0x7f6558138e40 con 0x7f655813f710 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.033+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65581d77a0 con 0x7f655813f710 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.033+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6548001b40 con 0x7f655813f710 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.033+0000 7f654effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.033+0000 7f654effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.033+0000 7f654effd640 10 monclient(hunting): dump: 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:12.033 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:12.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.033+0000 7f654effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:12.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.035+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660654982 0 0) 0x7f653c004eb0 con 0x7f65580a98f0 2024-03-25T19:46:12.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.035+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6558138e40 con 0x7f65580a98f0 2024-03-25T19:46:12.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.035+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6554001930 con 0x7f65580a98f0 2024-03-25T19:46:12.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.035+0000 7f654effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.035+0000 7f654effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.035+0000 7f654effd640 10 monclient(hunting): dump: 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:12.036 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:12.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.042+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378841518 0 0) 0x7f65580aba30 con 0x7f65581430c0 2024-03-25T19:46:12.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.042+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f653c004eb0 con 0x7f65581430c0 2024-03-25T19:46:12.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.042+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6550001a90 con 0x7f65581430c0 2024-03-25T19:46:12.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.042+0000 7f654effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.042+0000 7f654effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.042+0000 7f654effd640 10 monclient(hunting): dump: 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:12.043 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3357531652 0 0) 0x7f65581d77a0 con 0x7f655813f710 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 >> v1:172.21.15.53:6789/0 conn(0x7f65580a98f0 legacy=0x7f6558138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 >> v1:172.21.15.18:6789/0 conn(0x7f65581430c0 legacy=0x7f6558134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 1 monclient: found mon.c 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65581d8340 con 0x7f655813f710 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 10 monclient: _finish_auth 0 2024-03-25T19:46:12.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f654effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 5 monclient: authenticate success, global_id 4436 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 10 monclient: _renew_subs 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6558134820 con 0x7f655813f710 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 10 monclient: _renew_subs 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65581380c0 con 0x7f655813f710 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.048+0000 7f6560635640 1 librados: init done 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.049+0000 7f6560635640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.049+0000 7f6560635640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:12.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.049+0000 7f6560635640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:12.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.049+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65580627b0 con 0x7f655813f710 2024-03-25T19:46:12.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.386+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65480011f0 con 0x7f655813f710 2024-03-25T19:46:12.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.386+0000 7f654effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:12.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.386+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6548004c30 con 0x7f655813f710 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.386+0000 7f654effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.386+0000 7f654effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.386+0000 7f654effd640 10 monclient: dump: 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:12.387 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:12.388 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:12.388 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:12.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.386+0000 7f655cbce640 4 set_mon_vals no callback set 2024-03-25T19:46:12.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.387+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f654801ac40 con 0x7f655813f710 2024-03-25T19:46:12.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.388+0000 7f655e3d1640 10 monclient: get_auth_request con 0x7f653c06e060 auth_method 0 2024-03-25T19:46:12.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.388+0000 7f654effd640 10 client.4436.objecter ms_handle_connect 0x7f653c06e060 2024-03-25T19:46:12.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.392+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f65480548d0 con 0x7f655813f710 2024-03-25T19:46:12.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.393+0000 7f654effd640 10 client.4436.objecter ms_dispatch 0x7f6558000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:12.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.393+0000 7f654effd640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:12.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.393+0000 7f654effd640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:12.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.393+0000 7f654effd640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-25T19:46:12.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.393+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6548050210 con 0x7f655813f710 2024-03-25T19:46:12.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.393+0000 7f654effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:12.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.393+0000 7f654effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:12.647 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.647+0000 7f6560635640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.647 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.647+0000 7f6560635640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.647+0000 7f6560635640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:12.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.647+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f655814e880 con 0x7f655813f710 2024-03-25T19:46:12.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.687+0000 7f654effd640 1 -- 172.21.15.18:0/3814337750 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f6548052310 con 0x7f655813f710 2024-03-25T19:46:12.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.687+0000 7f654effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.687+0000 7f654effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:12.688 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:46:12.688 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:12.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 10 librados: watch_flush enter 2024-03-25T19:46:12.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 10 librados: watch_flush exit 2024-03-25T19:46:12.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 10 client.4436.objecter successfully canceled tick 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 >> v1:172.21.15.18:6800/1520000573 conn(0x7f653c06e060 legacy=0x7f653c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 10 monclient: shutdown 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 >> v1:172.21.15.18:6790/0 conn(0x7f655813f710 legacy=0x7f65581d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 shutdown_connections 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 >> 172.21.15.18:0/3814337750 conn(0x7f65580b7ea0 msgr2=0x7f6558074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 shutdown_connections 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 1 -- 172.21.15.18:0/3814337750 wait complete. 2024-03-25T19:46:12.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:12.690+0000 7f6560635640 1 librados: shutdown 2024-03-25T19:46:12.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:19.508 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:19.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.622+0000 7fa9d4354640 10 monclient: get_monmap_and_config 2024-03-25T19:46:19.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.622+0000 7fa9d4354640 10 monclient: build_initial_monmap 2024-03-25T19:46:19.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.622+0000 7fa9d4354640 10 monclient: monmap: 2024-03-25T19:46:19.622 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:19.622 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:19.622 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:46:19.623123+0000 2024-03-25T19:46:19.622 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:46:19.623123+0000 2024-03-25T19:46:19.622 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:19.623 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:19.623 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:19.623 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:46:19.623 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:46:19.623 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:19.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.622+0000 7fa9d4354640 1 Processor -- start 2024-03-25T19:46:19.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.623+0000 7fa9d4354640 1 -- start start 2024-03-25T19:46:19.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.623+0000 7fa9d4354640 10 monclient: init 2024-03-25T19:46:19.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:19.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:19.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 10 monclient(hunting): picked mon.noname-b con 0x7fa9cc0b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:19.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 10 monclient(hunting): picked mon.noname-c con 0x7fa9cc0b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:19.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 10 monclient(hunting): picked mon.noname-a con 0x7fa9cc14be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:19.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9cc0af940 con 0x7fa9cc14be90 2024-03-25T19:46:19.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9cc0b0a70 con 0x7fa9cc0b1730 2024-03-25T19:46:19.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9cc0b0d50 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:19.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.624+0000 7fa9d4354640 10 monclient(hunting): authenticate will time out at 5206.750961s 2024-03-25T19:46:19.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.633+0000 7fa9d28f1640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa9cc14be90 0x7fa9cc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:52676/0 (socket says 172.21.15.18:52676) 2024-03-25T19:46:19.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.633+0000 7fa9d28f1640 1 -- 172.21.15.18:0/3311969323 learned_addr learned my addr 172.21.15.18:0/3311969323 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:19.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.645+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217126009 0 0) 0x7fa9cc0af940 con 0x7fa9cc14be90 2024-03-25T19:46:19.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.645+0000 7fa9d10ee640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.645+0000 7fa9d10ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.645+0000 7fa9d10ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.645+0000 7fa9d10ee640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:46:19.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.646+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b8003520 con 0x7fa9cc14be90 2024-03-25T19:46:19.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.655+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345026698 0 0) 0x7fa9cc0b0d50 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.655+0000 7fa9d10ee640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.655+0000 7fa9d10ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.655+0000 7fa9d10ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.655+0000 7fa9d10ee640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:46:19.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.655+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc0af940 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.661+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506841642 0 0) 0x7fa9cc0b0a70 con 0x7fa9cc0b1730 2024-03-25T19:46:19.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.661+0000 7fa9d10ee640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.661+0000 7fa9d10ee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.661+0000 7fa9d10ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.661+0000 7fa9d10ee640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:46:19.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.661+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc0b0d50 con 0x7fa9cc0b1730 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 736955460 0 0) 0x7fa9cc0af940 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 >> v1:172.21.15.53:6789/0 conn(0x7fa9cc0b1730 legacy=0x7fa9cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 >> v1:172.21.15.18:6789/0 conn(0x7fa9cc14be90 legacy=0x7fa9cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 10 monclient: _finish_hunting 0 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 1 monclient: found mon.noname-b 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc0b1030 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 10 monclient: _finish_auth 0 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 10 monclient: _check_auth_tickets 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa9c8001670 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d10ee640 10 monclient: dump: 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d4354640 5 monclient: authenticate success, global_id 4439 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.667+0000 7fa9d4354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:19.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c80017d0 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d10ee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d10ee640 1 -- 172.21.15.18:0/3311969323 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa9c8001be0 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d4354640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d4354640 4 set_mon_vals no callback set 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d10ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d10ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d10ee640 10 monclient: dump: 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.686 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.687 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:19.687 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:19.687 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:19.687 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:19.687 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d4354640 10 monclient: shutdown 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d4354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.686+0000 7fa9d4354640 1 -- 172.21.15.18:0/3311969323 >> v1:172.21.15.18:6790/0 conn(0x7fa9cc0b4d60 legacy=0x7fa9cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.687+0000 7fa9d4354640 1 -- 172.21.15.18:0/3311969323 shutdown_connections 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.687+0000 7fa9d4354640 1 -- 172.21.15.18:0/3311969323 >> 172.21.15.18:0/3311969323 conn(0x7fa9cc0617a0 msgr2=0x7fa9cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.687+0000 7fa9d4354640 1 -- 172.21.15.18:0/3311969323 shutdown_connections 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.687+0000 7fa9d4354640 1 -- 172.21.15.18:0/3311969323 wait complete. 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.687+0000 7fa9d4354640 10 monclient: build_initial_monmap 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.687+0000 7fa9d4354640 10 monclient: monmap: 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 1 librados: starting msgr at 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 1 librados: starting objecter 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 1 Processor -- start 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 1 -- start start 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 1 librados: setting wanted keys 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 1 librados: calling monclient init 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 10 monclient: init 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 10 monclient(hunting): picked mon.noname-a con 0x7fa9cc0b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 10 monclient(hunting): picked mon.noname-b con 0x7fa9cc0b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.688+0000 7fa9d4354640 10 monclient(hunting): picked mon.noname-c con 0x7fa9cc14be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.689+0000 7fa9d4354640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9cc141e80 con 0x7fa9cc0b1730 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.689+0000 7fa9d4354640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9cc1d7570 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.689+0000 7fa9d4354640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9cc1d81a0 con 0x7fa9cc14be90 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.689+0000 7fa9d4354640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:19.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.689+0000 7fa9d4354640 10 monclient(hunting): authenticate will time out at 5206.815764s 2024-03-25T19:46:19.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.695+0000 7fa9d20f0640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa9cc0b1730 0x7fa9cc138220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:52724/0 (socket says 172.21.15.18:52724) 2024-03-25T19:46:19.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.695+0000 7fa9d20f0640 1 -- 172.21.15.18:0/2445264703 learned_addr learned my addr 172.21.15.18:0/2445264703 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:19.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.698+0000 7fa9c2ffd640 10 client.?.objecter ms_handle_connect 0x7fa9cc0b1730 2024-03-25T19:46:19.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.698+0000 7fa9c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.705+0000 7fa9c2ffd640 10 client.?.objecter ms_handle_connect 0x7fa9cc14be90 2024-03-25T19:46:19.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.705+0000 7fa9c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.713+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871371453 0 0) 0x7fa9cc141e80 con 0x7fa9cc0b1730 2024-03-25T19:46:19.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.713+0000 7fa9c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.713+0000 7fa9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.713+0000 7fa9c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.713+0000 7fa9c2ffd640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:46:19.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.713+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b0004eb0 con 0x7fa9cc0b1730 2024-03-25T19:46:19.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.720+0000 7fa9c2ffd640 10 client.?.objecter ms_handle_connect 0x7fa9cc0b4d60 2024-03-25T19:46:19.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.720+0000 7fa9c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.725+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234034499 0 0) 0x7fa9cc1d81a0 con 0x7fa9cc14be90 2024-03-25T19:46:19.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.725+0000 7fa9c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.725+0000 7fa9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.725+0000 7fa9c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.725+0000 7fa9c2ffd640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:46:19.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.725+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc141e80 con 0x7fa9cc14be90 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.729+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588136553 0 0) 0x7fa9cc1d7570 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.729+0000 7fa9c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.729+0000 7fa9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.729+0000 7fa9c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.729+0000 7fa9c2ffd640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.729+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc1d81a0 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.739+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2860396675 0 0) 0x7fa9cc1d81a0 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.739+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9cc1d7570 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.739+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa9bc001830 con 0x7fa9cc0b4d60 2024-03-25T19:46:19.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.739+0000 7fa9c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.739+0000 7fa9c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:19.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.739+0000 7fa9c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:46:19.739 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.739+0000 7fa9c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.740+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1188960357 0 0) 0x7fa9b0004eb0 con 0x7fa9cc0b1730 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.740+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9cc1d81a0 con 0x7fa9cc0b1730 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.740+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa9c8001670 con 0x7fa9cc0b1730 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.740+0000 7fa9c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.740+0000 7fa9c2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.740+0000 7fa9c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.751+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3132225229 0 0) 0x7fa9cc141e80 con 0x7fa9cc14be90 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.751+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b0004eb0 con 0x7fa9cc14be90 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.751+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa9c4001a70 con 0x7fa9cc14be90 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.751+0000 7fa9c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.751+0000 7fa9c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.751+0000 7fa9c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.751 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:19.752 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.752 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:19.752 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:19.752 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:19.752 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:19.752 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 849981275 0 0) 0x7fa9cc1d81a0 con 0x7fa9cc0b1730 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 >> v1:172.21.15.18:6790/0 conn(0x7fa9cc14be90 legacy=0x7fa9cc142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 >> v1:172.21.15.53:6789/0 conn(0x7fa9cc0b4d60 legacy=0x7fa9cc1d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 1 monclient: found mon.a 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc1d8330 con 0x7fa9cc0b1730 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:19.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 5 monclient: authenticate success, global_id 4473 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 10 monclient: _renew_subs 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc143110 con 0x7fa9cc0b1730 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 10 monclient: _renew_subs 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9cc142480 con 0x7fa9cc0b1730 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.758+0000 7fa9d4354640 1 librados: init done 2024-03-25T19:46:19.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.759+0000 7fa9d4354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:19.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.759+0000 7fa9d4354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:19.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.759+0000 7fa9d4354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:19.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:19.759+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9cc1401f0 con 0x7fa9cc0b1730 2024-03-25T19:46:20.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.109+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c80038a0 con 0x7fa9cc0b1730 2024-03-25T19:46:20.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.109+0000 7fa9c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:20.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.109+0000 7fa9d08ed640 4 set_mon_vals no callback set 2024-03-25T19:46:20.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.109+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa9c8003a90 con 0x7fa9cc0b1730 2024-03-25T19:46:20.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.109+0000 7fa9c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.109+0000 7fa9c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.109+0000 7fa9c2ffd640 10 monclient: dump: 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:20.110 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:20.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.110+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fa9c8003cb0 con 0x7fa9cc0b1730 2024-03-25T19:46:20.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.110+0000 7fa9d18ef640 10 monclient: get_auth_request con 0x7fa9b006e160 auth_method 0 2024-03-25T19:46:20.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.111+0000 7fa9c2ffd640 10 client.4473.objecter ms_handle_connect 0x7fa9b006e160 2024-03-25T19:46:20.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.115+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fa9c80581a0 con 0x7fa9cc0b1730 2024-03-25T19:46:20.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.115+0000 7fa9c2ffd640 10 client.4473.objecter ms_dispatch 0x7fa9cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:20.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.116+0000 7fa9c2ffd640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.116+0000 7fa9c2ffd640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.116+0000 7fa9c2ffd640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T19:46:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.116+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9c8050320 con 0x7fa9cc0b1730 2024-03-25T19:46:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.116+0000 7fa9c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:20.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.116+0000 7fa9c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:20.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.362+0000 7fa9d4354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:20.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.362+0000 7fa9d4354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:20.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.362+0000 7fa9d4354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:20.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.362+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9cc141cc0 con 0x7fa9cc0b1730 2024-03-25T19:46:20.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.418+0000 7fa9c2ffd640 1 -- 172.21.15.18:0/2445264703 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fa9c80547d0 con 0x7fa9cc0b1730 2024-03-25T19:46:20.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.418+0000 7fa9c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:20.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.418+0000 7fa9c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:20.418 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:46:20.418 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:20.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 10 librados: watch_flush enter 2024-03-25T19:46:20.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 10 librados: watch_flush exit 2024-03-25T19:46:20.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T19:46:20.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 10 client.4473.objecter successfully canceled tick 2024-03-25T19:46:20.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 >> v1:172.21.15.18:6800/1520000573 conn(0x7fa9b006e160 legacy=0x7fa9b0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 10 monclient: shutdown 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 >> v1:172.21.15.18:6789/0 conn(0x7fa9cc0b1730 legacy=0x7fa9cc138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 shutdown_connections 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 >> 172.21.15.18:0/2445264703 conn(0x7fa9cc0617a0 msgr2=0x7fa9cc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 shutdown_connections 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 1 -- 172.21.15.18:0/2445264703 wait complete. 2024-03-25T19:46:20.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:20.420+0000 7fa9d4354640 1 librados: shutdown 2024-03-25T19:46:20.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:27.238 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.349+0000 7fded804a640 10 monclient: get_monmap_and_config 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.349+0000 7fded804a640 10 monclient: build_initial_monmap 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.349+0000 7fded804a640 10 monclient: monmap: 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:46:27.350329+0000 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:46:27.350329+0000 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:27.349 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:27.350 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:27.350 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:46:27.350 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:46:27.350 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.350+0000 7fded804a640 1 Processor -- start 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.350+0000 7fded804a640 1 -- start start 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.350+0000 7fded804a640 10 monclient: init 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 10 monclient(hunting): picked mon.noname-c con 0x7fded00b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 10 monclient(hunting): picked mon.noname-a con 0x7fded00b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 10 monclient(hunting): picked mon.noname-b con 0x7fded014be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded00af940 con 0x7fded00b1730 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded00b0a70 con 0x7fded00b4d60 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded00b0d50 con 0x7fded014be90 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.351+0000 7fded804a640 10 monclient(hunting): authenticate will time out at 5214.478264s 2024-03-25T19:46:27.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.358+0000 7fded65e7640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fded014be90 0x7fded014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:42146/0 (socket says 172.21.15.18:42146) 2024-03-25T19:46:27.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.358+0000 7fded65e7640 1 -- 172.21.15.18:0/3368519855 learned_addr learned my addr 172.21.15.18:0/3368519855 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.369+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233438399 0 0) 0x7fded00b0d50 con 0x7fded014be90 2024-03-25T19:46:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.369+0000 7fded4de4640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.369+0000 7fded4de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.369+0000 7fded4de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.369+0000 7fded4de4640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:46:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.369+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdebc003520 con 0x7fded014be90 2024-03-25T19:46:27.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.382+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121170832 0 0) 0x7fded00b0a70 con 0x7fded00b4d60 2024-03-25T19:46:27.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.382+0000 7fded4de4640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.382+0000 7fded4de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.382+0000 7fded4de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.382+0000 7fded4de4640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:46:27.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.382+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded00b0d50 con 0x7fded00b4d60 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2645075157 0 0) 0x7fdebc003520 con 0x7fded014be90 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 >> v1:172.21.15.53:6789/0 conn(0x7fded00b4d60 legacy=0x7fded013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 >> v1:172.21.15.18:6789/0 conn(0x7fded00b1730 legacy=0x7fded00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 10 monclient: _finish_hunting 0 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 1 monclient: found mon.noname-b 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded00b1030 con 0x7fded014be90 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 10 monclient: _finish_auth 0 2024-03-25T19:46:27.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 10 monclient: _check_auth_tickets 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fdec8001a10 con 0x7fded014be90 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded4de4640 10 monclient: dump: 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:27.392 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:27.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded804a640 5 monclient: authenticate success, global_id 4445 2024-03-25T19:46:27.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.391+0000 7fded804a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:27.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.410+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdec8001b70 con 0x7fded014be90 2024-03-25T19:46:27.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.410+0000 7fded4de4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:27.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.410+0000 7fded4de4640 1 -- 172.21.15.18:0/3368519855 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fdec8001f80 con 0x7fded014be90 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.410+0000 7fded804a640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.410+0000 7fded804a640 4 set_mon_vals no callback set 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.410+0000 7fded4de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.410+0000 7fded4de4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.410+0000 7fded4de4640 10 monclient: dump: 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:27.411 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.411+0000 7fded804a640 10 monclient: shutdown 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.411+0000 7fded804a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.411+0000 7fded804a640 1 -- 172.21.15.18:0/3368519855 >> v1:172.21.15.18:6790/0 conn(0x7fded014be90 legacy=0x7fded014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.411+0000 7fded804a640 1 -- 172.21.15.18:0/3368519855 shutdown_connections 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.411+0000 7fded804a640 1 -- 172.21.15.18:0/3368519855 >> 172.21.15.18:0/3368519855 conn(0x7fded00617a0 msgr2=0x7fded0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.412+0000 7fded804a640 1 -- 172.21.15.18:0/3368519855 shutdown_connections 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.412+0000 7fded804a640 1 -- 172.21.15.18:0/3368519855 wait complete. 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.412+0000 7fded804a640 10 monclient: build_initial_monmap 2024-03-25T19:46:27.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.413+0000 7fded804a640 10 monclient: monmap: 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.413+0000 7fded804a640 1 librados: starting msgr at 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.413+0000 7fded804a640 1 librados: starting objecter 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.413+0000 7fded804a640 1 Processor -- start 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.413+0000 7fded804a640 1 -- start start 2024-03-25T19:46:27.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.413+0000 7fded804a640 1 librados: setting wanted keys 2024-03-25T19:46:27.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.413+0000 7fded804a640 1 librados: calling monclient init 2024-03-25T19:46:27.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.413+0000 7fded804a640 10 monclient: init 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 10 monclient(hunting): picked mon.noname-a con 0x7fded00b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 10 monclient(hunting): picked mon.noname-c con 0x7fded00b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 10 monclient(hunting): picked mon.noname-b con 0x7fded014be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded0141650 con 0x7fded00b1730 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded0141930 con 0x7fded014be90 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded0141c10 con 0x7fded00b4d60 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:27.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.414+0000 7fded804a640 10 monclient(hunting): authenticate will time out at 5214.541384s 2024-03-25T19:46:27.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.422+0000 7fded5de6640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fded00b1730 0x7fded01df160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:38062/0 (socket says 172.21.15.18:38062) 2024-03-25T19:46:27.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.422+0000 7fded5de6640 1 -- 172.21.15.18:0/2198082184 learned_addr learned my addr 172.21.15.18:0/2198082184 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:27.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.423+0000 7fdec67fc640 10 client.?.objecter ms_handle_connect 0x7fded00b1730 2024-03-25T19:46:27.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.423+0000 7fdec67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:27.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.428+0000 7fdec67fc640 10 client.?.objecter ms_handle_connect 0x7fded014be90 2024-03-25T19:46:27.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.428+0000 7fdec67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:27.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.429+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559637544 0 0) 0x7fded0141650 con 0x7fded00b1730 2024-03-25T19:46:27.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.429+0000 7fdec67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.429+0000 7fdec67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.429+0000 7fdec67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.429+0000 7fdec67fc640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:46:27.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.429+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4004eb0 con 0x7fded00b1730 2024-03-25T19:46:27.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.432+0000 7fdec67fc640 10 client.?.objecter ms_handle_connect 0x7fded00b4d60 2024-03-25T19:46:27.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.432+0000 7fdec67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:27.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.439+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533060684 0 0) 0x7fded0141930 con 0x7fded014be90 2024-03-25T19:46:27.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.439+0000 7fdec67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.439+0000 7fdec67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.439+0000 7fdec67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.439+0000 7fdec67fc640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:46:27.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.439+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded0141650 con 0x7fded014be90 2024-03-25T19:46:27.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.441+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1746401657 0 0) 0x7fdeb4004eb0 con 0x7fded00b1730 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.441+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fded0141930 con 0x7fded00b1730 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.441+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fdecc001a10 con 0x7fded00b1730 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.441+0000 7fdec67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.441+0000 7fdec67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.441+0000 7fdec67fc640 10 monclient(hunting): dump: 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:27.441 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:27.442 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:27.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.441+0000 7fdec67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:27.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.443+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547817337 0 0) 0x7fded0141c10 con 0x7fded00b4d60 2024-03-25T19:46:27.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.443+0000 7fdec67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.443+0000 7fdec67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.443+0000 7fdec67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.443+0000 7fdec67fc640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:46:27.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.443+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4004eb0 con 0x7fded00b4d60 2024-03-25T19:46:27.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 214844139 0 0) 0x7fded0141930 con 0x7fded00b1730 2024-03-25T19:46:27.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 >> v1:172.21.15.18:6790/0 conn(0x7fded00b4d60 legacy=0x7fded01df870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 >> v1:172.21.15.53:6789/0 conn(0x7fded014be90 legacy=0x7fded01e6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:27.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 1 monclient: found mon.a 2024-03-25T19:46:27.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:27.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:27.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded0137fd0 con 0x7fded00b1730 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fdec67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 5 monclient: authenticate success, global_id 4479 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 10 monclient: _renew_subs 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded0140830 con 0x7fded00b1730 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 10 monclient: _renew_subs 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fded0135720 con 0x7fded00b1730 2024-03-25T19:46:27.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.450+0000 7fded804a640 1 librados: init done 2024-03-25T19:46:27.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.451+0000 7fded804a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:27.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.451+0000 7fded804a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:27.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.451+0000 7fded804a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:27.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.451+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded00627b0 con 0x7fded00b1730 2024-03-25T19:46:27.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.479+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdecc001c50 con 0x7fded00b1730 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.479+0000 7fdec67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.479+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fdecc003a60 con 0x7fded00b1730 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.479+0000 7fdec67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.479+0000 7fdec67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.479+0000 7fdec67fc640 10 monclient: dump: 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:27.481 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.482 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:27.482 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:27.482 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:27.482 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:27.482 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:27.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.479+0000 7fdec7fff640 4 set_mon_vals no callback set 2024-03-25T19:46:27.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.479+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fdecc003c80 con 0x7fded00b1730 2024-03-25T19:46:27.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.480+0000 7fded55e5640 10 monclient: get_auth_request con 0x7fdeb406ddb0 auth_method 0 2024-03-25T19:46:27.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.480+0000 7fdec67fc640 10 client.4479.objecter ms_handle_connect 0x7fdeb406ddb0 2024-03-25T19:46:27.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.482+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fdecc0548d0 con 0x7fded00b1730 2024-03-25T19:46:27.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.482+0000 7fdec67fc640 10 client.4479.objecter ms_dispatch 0x7fded0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:27.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.482+0000 7fdec67fc640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:27.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.482+0000 7fdec67fc640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:27.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.482+0000 7fdec67fc640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T19:46:27.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.814+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdecc0503e0 con 0x7fded00b1730 2024-03-25T19:46:27.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.814+0000 7fdec67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:27.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:27.814+0000 7fdec67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:27.914 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:55.444 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.922 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:55.452 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.961 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:55.492 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.969 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:55.499 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.980 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:55.511 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:27.997 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:55.493 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:28.000 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:55.497 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:28.031 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:55.562 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:28.034 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:05:55.530 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:28.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.034+0000 7fded804a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.034+0000 7fded804a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.034+0000 7fded804a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:28.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.034+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fded01409c0 con 0x7fded00b1730 2024-03-25T19:46:28.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.162+0000 7fdec67fc640 1 -- 172.21.15.18:0/2198082184 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fdecc052310 con 0x7fded00b1730 2024-03-25T19:46:28.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.162+0000 7fdec67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.162+0000 7fdec67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:28.162 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:46:28.163 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:28.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 10 librados: watch_flush enter 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 10 librados: watch_flush exit 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 10 client.4479.objecter successfully canceled tick 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 >> v1:172.21.15.18:6800/1520000573 conn(0x7fdeb406ddb0 legacy=0x7fdeb4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 10 monclient: shutdown 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 >> v1:172.21.15.18:6789/0 conn(0x7fded00b1730 legacy=0x7fded01df160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 shutdown_connections 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 >> 172.21.15.18:0/2198082184 conn(0x7fded00617a0 msgr2=0x7fded013c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 shutdown_connections 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 1 -- 172.21.15.18:0/2198082184 wait complete. 2024-03-25T19:46:28.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:28.164+0000 7fded804a640 1 librados: shutdown 2024-03-25T19:46:28.174 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:34.981 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.093+0000 7f97d2ddb640 10 monclient: get_monmap_and_config 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.093+0000 7f97d2ddb640 10 monclient: build_initial_monmap 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.093+0000 7f97d2ddb640 10 monclient: monmap: 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:46:35.094941+0000 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:46:35.094941+0000 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:35.094 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:35.095 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:46:35.095 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:46:35.095 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.094+0000 7f97d2ddb640 1 Processor -- start 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.094+0000 7f97d2ddb640 1 -- start start 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.094+0000 7f97d2ddb640 10 monclient: init 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.094+0000 7f97d2ddb640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.094+0000 7f97d2ddb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.095+0000 7f97d2ddb640 10 monclient(hunting): picked mon.noname-a con 0x7f97cc0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.095+0000 7f97d2ddb640 10 monclient(hunting): picked mon.noname-c con 0x7f97cc13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.095+0000 7f97d2ddb640 10 monclient(hunting): picked mon.noname-b con 0x7f97cc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.095+0000 7f97d2ddb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc142520 con 0x7f97cc0a98f0 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.095+0000 7f97d2ddb640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc0b28a0 con 0x7f97cc13f710 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.095+0000 7f97d2ddb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc0b2a30 con 0x7f97cc1430c0 2024-03-25T19:46:35.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.095+0000 7f97d2ddb640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:35.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.095+0000 7f97d2ddb640 10 monclient(hunting): authenticate will time out at 5222.222820s 2024-03-25T19:46:35.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.108+0000 7f97d0b77640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f97cc0a98f0 0x7f97cc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:42650/0 (socket says 172.21.15.18:42650) 2024-03-25T19:46:35.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.108+0000 7f97d0b77640 1 -- 172.21.15.18:0/966914880 learned_addr learned my addr 172.21.15.18:0/966914880 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:35.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.124+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867235962 0 0) 0x7f97cc0b28a0 con 0x7f97cc13f710 2024-03-25T19:46:35.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.124+0000 7f97cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.124+0000 7f97cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.124+0000 7f97cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.124+0000 7f97cb7fe640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:46:35.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.124+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97bc003520 con 0x7f97cc13f710 2024-03-25T19:46:35.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.130+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490262074 0 0) 0x7f97cc142520 con 0x7f97cc0a98f0 2024-03-25T19:46:35.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.130+0000 7f97cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.130+0000 7f97cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.130+0000 7f97cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.130+0000 7f97cb7fe640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:46:35.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.130+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97cc0b28a0 con 0x7f97cc0a98f0 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1837630283 0 0) 0x7f97bc003520 con 0x7f97cc13f710 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 >> v1:172.21.15.18:6790/0 conn(0x7f97cc1430c0 legacy=0x7f97cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 >> v1:172.21.15.18:6789/0 conn(0x7f97cc0a98f0 legacy=0x7f97cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 1 monclient: found mon.noname-c 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc0b44c0 con 0x7f97cc13f710 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f97b8001690 con 0x7f97cc13f710 2024-03-25T19:46:35.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97d2ddb640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97d2ddb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97cb7fe640 10 monclient: dump: 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:35.138 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:35.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.135+0000 7f97d2ddb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:35.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.153+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97b80017f0 con 0x7f97cc13f710 2024-03-25T19:46:35.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.153+0000 7f97cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:35.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.153+0000 7f97cb7fe640 1 -- 172.21.15.18:0/966914880 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f97b8001c00 con 0x7f97cc13f710 2024-03-25T19:46:35.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.153+0000 7f97cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.154+0000 7f97cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.154+0000 7f97cb7fe640 10 monclient: dump: 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.154+0000 7f97d2ddb640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.154+0000 7f97d2ddb640 4 set_mon_vals no callback set 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.154+0000 7f97d2ddb640 10 monclient: shutdown 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.154+0000 7f97d2ddb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.154+0000 7f97d2ddb640 1 -- 172.21.15.18:0/966914880 >> v1:172.21.15.53:6789/0 conn(0x7f97cc13f710 legacy=0x7f97cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.155+0000 7f97d2ddb640 1 -- 172.21.15.18:0/966914880 shutdown_connections 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.155+0000 7f97d2ddb640 1 -- 172.21.15.18:0/966914880 >> 172.21.15.18:0/966914880 conn(0x7f97cc0b7ea0 msgr2=0x7f97cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.155+0000 7f97d2ddb640 1 -- 172.21.15.18:0/966914880 shutdown_connections 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.155+0000 7f97d2ddb640 1 -- 172.21.15.18:0/966914880 wait complete. 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.155+0000 7f97d2ddb640 10 monclient: build_initial_monmap 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.155+0000 7f97d2ddb640 10 monclient: monmap: 2024-03-25T19:46:35.157 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.156+0000 7f97d2ddb640 1 librados: starting msgr at 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.156+0000 7f97d2ddb640 1 librados: starting objecter 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.156+0000 7f97d2ddb640 1 Processor -- start 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.156+0000 7f97d2ddb640 1 -- start start 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.156+0000 7f97d2ddb640 1 librados: setting wanted keys 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.156+0000 7f97d2ddb640 1 librados: calling monclient init 2024-03-25T19:46:35.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.156+0000 7f97d2ddb640 10 monclient: init 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 10 monclient(hunting): picked mon.noname-a con 0x7f97cc0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 10 monclient(hunting): picked mon.noname-b con 0x7f97cc13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 10 monclient(hunting): picked mon.noname-c con 0x7f97cc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc0abdc0 con 0x7f97cc0a98f0 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc1d7730 con 0x7f97cc13f710 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc138f10 con 0x7f97cc1430c0 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:35.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.157+0000 7f97d2ddb640 10 monclient(hunting): authenticate will time out at 5222.285040s 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.166+0000 7f97d1378640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f97cc1430c0 0x7f97cc1d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46812/0 (socket says 172.21.15.18:46812) 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.166+0000 7f97d1378640 1 -- 172.21.15.18:0/1464512061 learned_addr learned my addr 172.21.15.18:0/1464512061 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:35.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.168+0000 7f97c97fa640 10 client.?.objecter ms_handle_connect 0x7f97cc1430c0 2024-03-25T19:46:35.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.168+0000 7f97c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.176+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913865560 0 0) 0x7f97cc138f10 con 0x7f97cc1430c0 2024-03-25T19:46:35.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.176+0000 7f97c97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.176+0000 7f97c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.176+0000 7f97c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.176+0000 7f97c97fa640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:46:35.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.176+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b0004eb0 con 0x7f97cc1430c0 2024-03-25T19:46:35.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.182+0000 7f97c97fa640 10 client.?.objecter ms_handle_connect 0x7f97cc13f710 2024-03-25T19:46:35.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.182+0000 7f97c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.184+0000 7f97c97fa640 10 client.?.objecter ms_handle_connect 0x7f97cc0a98f0 2024-03-25T19:46:35.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.184+0000 7f97c97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.186+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3884776441 0 0) 0x7f97b0004eb0 con 0x7f97cc1430c0 2024-03-25T19:46:35.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.186+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97cc138f10 con 0x7f97cc1430c0 2024-03-25T19:46:35.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.200+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f97c4001d10 con 0x7f97cc1430c0 2024-03-25T19:46:35.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.200+0000 7f97c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.200+0000 7f97c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:35.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.200+0000 7f97c97fa640 10 monclient(hunting): dump: 2024-03-25T19:46:35.201 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:35.201 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:35.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.200+0000 7f97c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:35.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.205+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575194864 0 0) 0x7f97cc1d7730 con 0x7f97cc13f710 2024-03-25T19:46:35.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.205+0000 7f97c97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.205+0000 7f97c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.205+0000 7f97c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.205+0000 7f97c97fa640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:46:35.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.205+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b0004eb0 con 0x7f97cc13f710 2024-03-25T19:46:35.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3573493152 0 0) 0x7f97cc138f10 con 0x7f97cc1430c0 2024-03-25T19:46:35.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 >> v1:172.21.15.53:6789/0 conn(0x7f97cc13f710 legacy=0x7f97cc1d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 >> v1:172.21.15.18:6789/0 conn(0x7f97cc0a98f0 legacy=0x7f97cc138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:35.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 1 monclient: found mon.c 2024-03-25T19:46:35.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:35.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc1390a0 con 0x7f97cc1430c0 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 5 monclient: authenticate success, global_id 4454 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 10 monclient: _renew_subs 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97cc1d8320 con 0x7f97cc1430c0 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 10 monclient: _renew_subs 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97cc1d43c0 con 0x7f97cc1430c0 2024-03-25T19:46:35.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.207+0000 7f97d2ddb640 1 librados: init done 2024-03-25T19:46:35.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.209+0000 7f97d2ddb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:35.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.209+0000 7f97d2ddb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:35.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.209+0000 7f97d2ddb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:35.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.209+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97cc0627b0 con 0x7f97cc1430c0 2024-03-25T19:46:35.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.537+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97c4003970 con 0x7f97cc1430c0 2024-03-25T19:46:35.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.537+0000 7f97c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:35.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.537+0000 7f97caffd640 4 set_mon_vals no callback set 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.537+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f97c4003d40 con 0x7f97cc1430c0 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.537+0000 7f97c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.537+0000 7f97c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.537+0000 7f97c97fa640 10 monclient: dump: 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:35.539 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:35.540 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:35.540 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:35.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.538+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f97c4003f60 con 0x7f97cc1430c0 2024-03-25T19:46:35.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.539+0000 7f97d0b77640 10 monclient: get_auth_request con 0x7f97b006dd40 auth_method 0 2024-03-25T19:46:35.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.539+0000 7f97c97fa640 10 client.4454.objecter ms_handle_connect 0x7f97b006dd40 2024-03-25T19:46:35.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.546+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f97c4054ac0 con 0x7f97cc1430c0 2024-03-25T19:46:35.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.546+0000 7f97c97fa640 10 client.4454.objecter ms_dispatch 0x7f97cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.546+0000 7f97c97fa640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.546+0000 7f97c97fa640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.546+0000 7f97c97fa640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T19:46:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.546+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97c40505d0 con 0x7f97cc1430c0 2024-03-25T19:46:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.546+0000 7f97c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:35.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.546+0000 7f97c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:35.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.803+0000 7f97d2ddb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:35.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.803+0000 7f97d2ddb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:35.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.803+0000 7f97d2ddb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:35.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.803+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97cc069f40 con 0x7f97cc1430c0 2024-03-25T19:46:35.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.838+0000 7f97c97fa640 1 -- 172.21.15.18:0/1464512061 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f97c4052500 con 0x7f97cc1430c0 2024-03-25T19:46:35.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.838+0000 7f97c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:35.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.838+0000 7f97c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:35.839 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:46:35.839 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.840+0000 7f97d2ddb640 10 librados: watch_flush enter 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.840+0000 7f97d2ddb640 10 librados: watch_flush exit 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.840+0000 7f97d2ddb640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.840+0000 7f97d2ddb640 10 client.4454.objecter successfully canceled tick 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 >> v1:172.21.15.18:6800/1520000573 conn(0x7f97b006dd40 legacy=0x7f97b0070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 10 monclient: shutdown 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 >> v1:172.21.15.18:6790/0 conn(0x7f97cc1430c0 legacy=0x7f97cc1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 shutdown_connections 2024-03-25T19:46:35.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 >> 172.21.15.18:0/1464512061 conn(0x7f97cc0b7ea0 msgr2=0x7f97cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:35.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 shutdown_connections 2024-03-25T19:46:35.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 1 -- 172.21.15.18:0/1464512061 wait complete. 2024-03-25T19:46:35.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:35.841+0000 7f97d2ddb640 1 librados: shutdown 2024-03-25T19:46:35.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:42.661 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.778+0000 7fc904e1e640 10 monclient: get_monmap_and_config 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.778+0000 7fc904e1e640 10 monclient: build_initial_monmap 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.778+0000 7fc904e1e640 10 monclient: monmap: 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:46:42.779993+0000 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:46:42.779993+0000 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:42.779 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:42.780 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:46:42.780 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:46:42.780 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:42.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.778+0000 7fc904e1e640 1 Processor -- start 2024-03-25T19:46:42.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.778+0000 7fc904e1e640 1 -- start start 2024-03-25T19:46:42.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.778+0000 7fc904e1e640 10 monclient: init 2024-03-25T19:46:42.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:42.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:42.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9000a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:42.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 10 monclient(hunting): picked mon.noname-c con 0x7fc90013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:42.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 10 monclient(hunting): picked mon.noname-b con 0x7fc9001430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:42.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900142520 con 0x7fc9000a98f0 2024-03-25T19:46:42.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b28a0 con 0x7fc90013f710 2024-03-25T19:46:42.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b2a30 con 0x7fc9001430c0 2024-03-25T19:46:42.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:42.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.779+0000 7fc904e1e640 10 monclient(hunting): authenticate will time out at 5229.907533s 2024-03-25T19:46:42.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.783+0000 7fc8fdd9b640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc90013f710 0x7fc9001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:42730/0 (socket says 172.21.15.18:42730) 2024-03-25T19:46:42.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.783+0000 7fc8fdd9b640 1 -- 172.21.15.18:0/3250146319 learned_addr learned my addr 172.21.15.18:0/3250146319 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:42.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.801+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483931322 0 0) 0x7fc9000b28a0 con 0x7fc90013f710 2024-03-25T19:46:42.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.801+0000 7fc8fd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.801+0000 7fc8fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.801+0000 7fc8fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.801+0000 7fc8fd59a640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:46:42.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.801+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e8003520 con 0x7fc90013f710 2024-03-25T19:46:42.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.806+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936112862 0 0) 0x7fc9000b2a30 con 0x7fc9001430c0 2024-03-25T19:46:42.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.806+0000 7fc8fd59a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.806+0000 7fc8fd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.806+0000 7fc8fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.806+0000 7fc8fd59a640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:46:42.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.806+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9000b28a0 con 0x7fc9001430c0 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2171742364 0 0) 0x7fc8e8003520 con 0x7fc90013f710 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 >> v1:172.21.15.18:6790/0 conn(0x7fc9001430c0 legacy=0x7fc90014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 >> v1:172.21.15.18:6789/0 conn(0x7fc9000a98f0 legacy=0x7fc9000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 10 monclient: _finish_hunting 0 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 1 monclient: found mon.noname-c 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000b44c0 con 0x7fc90013f710 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 10 monclient: _finish_auth 0 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 10 monclient: _check_auth_tickets 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc8f4001690 con 0x7fc90013f710 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc8fd59a640 10 monclient: dump: 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:42.814 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:42.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc904e1e640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:46:42.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.812+0000 7fc904e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.836+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8f40017f0 con 0x7fc90013f710 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.836+0000 7fc8fd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.836+0000 7fc8fd59a640 1 -- 172.21.15.18:0/3250146319 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc8f4001c00 con 0x7fc90013f710 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.836+0000 7fc8fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.836+0000 7fc8fd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.836+0000 7fc8fd59a640 10 monclient: dump: 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.836+0000 7fc904e1e640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:42.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.836+0000 7fc904e1e640 4 set_mon_vals no callback set 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.837+0000 7fc904e1e640 10 monclient: shutdown 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.837+0000 7fc904e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.837+0000 7fc904e1e640 1 -- 172.21.15.18:0/3250146319 >> v1:172.21.15.53:6789/0 conn(0x7fc90013f710 legacy=0x7fc9001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.837+0000 7fc904e1e640 1 -- 172.21.15.18:0/3250146319 shutdown_connections 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.837+0000 7fc904e1e640 1 -- 172.21.15.18:0/3250146319 >> 172.21.15.18:0/3250146319 conn(0x7fc9000b7ea0 msgr2=0x7fc9000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.837+0000 7fc904e1e640 1 -- 172.21.15.18:0/3250146319 shutdown_connections 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.837+0000 7fc904e1e640 1 -- 172.21.15.18:0/3250146319 wait complete. 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.838+0000 7fc904e1e640 10 monclient: build_initial_monmap 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.838+0000 7fc904e1e640 10 monclient: monmap: 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:42.840 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 librados: starting msgr at 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 librados: starting objecter 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 Processor -- start 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 -- start start 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 librados: setting wanted keys 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 librados: calling monclient init 2024-03-25T19:46:42.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 10 monclient: init 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 10 monclient(hunting): picked mon.noname-a con 0x7fc9000a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 10 monclient(hunting): picked mon.noname-b con 0x7fc90013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 10 monclient(hunting): picked mon.noname-c con 0x7fc9001430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9001354c0 con 0x7fc9000a98f0 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9001319b0 con 0x7fc90013f710 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900135070 con 0x7fc9001430c0 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.839+0000 7fc904e1e640 10 monclient(hunting): authenticate will time out at 5229.967728s 2024-03-25T19:46:42.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.841+0000 7fc8fed9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc9001430c0 0x7fc9001312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:60582/0 (socket says 172.21.15.18:60582) 2024-03-25T19:46:42.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.841+0000 7fc8fed9d640 1 -- 172.21.15.18:0/2609013136 learned_addr learned my addr 172.21.15.18:0/2609013136 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:42.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.847+0000 7fc8eeffd640 10 client.?.objecter ms_handle_connect 0x7fc90013f710 2024-03-25T19:46:42.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.847+0000 7fc8eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:42.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.853+0000 7fc8eeffd640 10 client.?.objecter ms_handle_connect 0x7fc9001430c0 2024-03-25T19:46:42.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.853+0000 7fc8eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:42.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.857+0000 7fc8eeffd640 10 client.?.objecter ms_handle_connect 0x7fc9000a98f0 2024-03-25T19:46:42.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.857+0000 7fc8eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:42.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.859+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887356453 0 0) 0x7fc9001319b0 con 0x7fc90013f710 2024-03-25T19:46:42.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.859+0000 7fc8eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.859+0000 7fc8eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.859+0000 7fc8eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.859+0000 7fc8eeffd640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:46:42.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.860+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e0004eb0 con 0x7fc90013f710 2024-03-25T19:46:42.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.863+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316482305 0 0) 0x7fc900135070 con 0x7fc9001430c0 2024-03-25T19:46:42.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.863+0000 7fc8eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.863+0000 7fc8eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.863+0000 7fc8eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.863+0000 7fc8eeffd640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:46:42.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.863+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9001319b0 con 0x7fc9001430c0 2024-03-25T19:46:42.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.873+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 611099629 0 0) 0x7fc9001319b0 con 0x7fc9001430c0 2024-03-25T19:46:42.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.873+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc900135070 con 0x7fc9001430c0 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.873+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc8f8001c50 con 0x7fc9001430c0 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.873+0000 7fc8eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.873+0000 7fc8eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.873+0000 7fc8eeffd640 10 monclient(hunting): dump: 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:42.875 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:42.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.873+0000 7fc8eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:42.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.883+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031721497 0 0) 0x7fc8e0004eb0 con 0x7fc90013f710 2024-03-25T19:46:42.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.883+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9001319b0 con 0x7fc90013f710 2024-03-25T19:46:42.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.883+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc8f4001b00 con 0x7fc90013f710 2024-03-25T19:46:42.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.883+0000 7fc8eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.883+0000 7fc8eeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:42.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.883+0000 7fc8eeffd640 10 monclient(hunting): dump: 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:42.885 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:42.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2352292642 0 0) 0x7fc900135070 con 0x7fc9001430c0 2024-03-25T19:46:42.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 >> v1:172.21.15.53:6789/0 conn(0x7fc90013f710 legacy=0x7fc900130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 >> v1:172.21.15.18:6789/0 conn(0x7fc9000a98f0 legacy=0x7fc9001d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:42.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 1 monclient: found mon.c 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900135200 con 0x7fc9001430c0 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc8eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 5 monclient: authenticate success, global_id 4460 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 10 monclient: _renew_subs 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9000b7010 con 0x7fc9001430c0 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 10 monclient: _renew_subs 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90013b890 con 0x7fc9001430c0 2024-03-25T19:46:42.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.885+0000 7fc904e1e640 1 librados: init done 2024-03-25T19:46:42.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.886+0000 7fc904e1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:42.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.886+0000 7fc904e1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:42.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.886+0000 7fc904e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:42.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:42.886+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900137160 con 0x7fc9001430c0 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.214+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8f80020e0 con 0x7fc9001430c0 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.214+0000 7fc8eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.214+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc8f8003bf0 con 0x7fc9001430c0 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.214+0000 7fc8eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.215+0000 7fc8eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.215+0000 7fc8eeffd640 10 monclient: dump: 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:43.216 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:43.217 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:43.217 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:43.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.214+0000 7fc8fcd99640 4 set_mon_vals no callback set 2024-03-25T19:46:43.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.215+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fc8f8003e10 con 0x7fc9001430c0 2024-03-25T19:46:43.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.216+0000 7fc8fe59c640 10 monclient: get_auth_request con 0x7fc8e006ded0 auth_method 0 2024-03-25T19:46:43.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.216+0000 7fc8eeffd640 10 client.4460.objecter ms_handle_connect 0x7fc8e006ded0 2024-03-25T19:46:43.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.222+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fc8f8054a30 con 0x7fc9001430c0 2024-03-25T19:46:43.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.222+0000 7fc8eeffd640 10 client.4460.objecter ms_dispatch 0x7fc900000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:43.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.222+0000 7fc8eeffd640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:43.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.222+0000 7fc8eeffd640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:43.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.222+0000 7fc8eeffd640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-25T19:46:43.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.222+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8f8050540 con 0x7fc9001430c0 2024-03-25T19:46:43.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.222+0000 7fc8eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:43.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.222+0000 7fc8eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:43.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.471+0000 7fc904e1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:43.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.471+0000 7fc904e1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:43.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.471+0000 7fc904e1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:46:43.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.471+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc900135670 con 0x7fc9001430c0 2024-03-25T19:46:43.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.510+0000 7fc8eeffd640 1 -- 172.21.15.18:0/2609013136 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fc900135670 con 0x7fc9001430c0 2024-03-25T19:46:43.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.510+0000 7fc8eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:43.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.510+0000 7fc8eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:43.511 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:46:43.511 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:43.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 10 librados: watch_flush enter 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 10 librados: watch_flush exit 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 10 client.4460.objecter successfully canceled tick 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 >> v1:172.21.15.18:6800/1520000573 conn(0x7fc8e006ded0 legacy=0x7fc8e0070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 10 monclient: shutdown 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 >> v1:172.21.15.18:6790/0 conn(0x7fc9001430c0 legacy=0x7fc9001312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 shutdown_connections 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 >> 172.21.15.18:0/2609013136 conn(0x7fc9000b7ea0 msgr2=0x7fc90013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 shutdown_connections 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 1 -- 172.21.15.18:0/2609013136 wait complete. 2024-03-25T19:46:43.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:43.512+0000 7fc904e1e640 1 librados: shutdown 2024-03-25T19:46:43.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:48.001 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:15.531 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.018 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:15.548 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.051 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:15.581 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.060 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:15.591 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.072 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:15.603 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.075 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:06:15.571 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.081 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:06:15.577 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.105 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:15.636 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:48.124 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:06:15.620 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:50.331 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:50.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.445+0000 7f88ab231640 10 monclient: get_monmap_and_config 2024-03-25T19:46:50.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.445+0000 7f88ab231640 10 monclient: build_initial_monmap 2024-03-25T19:46:50.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.445+0000 7f88ab231640 10 monclient: monmap: 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:46:50.447716+0000 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:46:50.447716+0000 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:46:50.447 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.446+0000 7f88ab231640 1 Processor -- start 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.446+0000 7f88ab231640 1 -- start start 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 10 monclient: init 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 10 monclient(hunting): picked mon.noname-b con 0x7f88a40b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 10 monclient(hunting): picked mon.noname-c con 0x7f88a40b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 10 monclient(hunting): picked mon.noname-a con 0x7f88a414be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a40af940 con 0x7f88a414be90 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a40b0a70 con 0x7f88a40b1730 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a40b0d50 con 0x7f88a40b4d60 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.447+0000 7f88ab231640 10 monclient(hunting): authenticate will time out at 5237.575562s 2024-03-25T19:46:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.457+0000 7f88a3fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f88a40b1730 0x7f88a40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:48868/0 (socket says 172.21.15.18:48868) 2024-03-25T19:46:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.457+0000 7f88a3fff640 1 -- 172.21.15.18:0/4287872210 learned_addr learned my addr 172.21.15.18:0/4287872210 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.467+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040478840 0 0) 0x7f88a40b0d50 con 0x7f88a40b4d60 2024-03-25T19:46:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.467+0000 7f88a37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.467+0000 7f88a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.467+0000 7f88a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.467+0000 7f88a37fe640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:46:50.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.467+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8894003520 con 0x7f88a40b4d60 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3762413860 0 0) 0x7f8894003520 con 0x7f88a40b4d60 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 >> v1:172.21.15.53:6789/0 conn(0x7f88a40b1730 legacy=0x7f88a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 >> v1:172.21.15.18:6789/0 conn(0x7f88a414be90 legacy=0x7f88a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 1 monclient: found mon.noname-b 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a40b1030 con 0x7f88a40b4d60 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:50.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8898001690 con 0x7f88a40b4d60 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88a37fe640 10 monclient: dump: 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:50.478 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:50.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88ab231640 5 monclient: authenticate success, global_id 4463 2024-03-25T19:46:50.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.476+0000 7f88ab231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.491+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88980017f0 con 0x7f88a40b4d60 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.491+0000 7f88a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.491+0000 7f88a37fe640 1 -- 172.21.15.18:0/4287872210 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8898001c00 con 0x7f88a40b4d60 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.491+0000 7f88a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.491+0000 7f88a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.491+0000 7f88a37fe640 10 monclient: dump: 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:50.492 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:50.493 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:50.493 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:50.493 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:50.493 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:50.493 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:50.493 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:50.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.491+0000 7f88ab231640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:50.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.491+0000 7f88ab231640 4 set_mon_vals no callback set 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.492+0000 7f88ab231640 10 monclient: shutdown 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.492+0000 7f88ab231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.492+0000 7f88ab231640 1 -- 172.21.15.18:0/4287872210 >> v1:172.21.15.18:6790/0 conn(0x7f88a40b4d60 legacy=0x7f88a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.492+0000 7f88ab231640 1 -- 172.21.15.18:0/4287872210 shutdown_connections 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.492+0000 7f88ab231640 1 -- 172.21.15.18:0/4287872210 >> 172.21.15.18:0/4287872210 conn(0x7f88a40617a0 msgr2=0x7f88a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.492+0000 7f88ab231640 1 -- 172.21.15.18:0/4287872210 shutdown_connections 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.492+0000 7f88ab231640 1 -- 172.21.15.18:0/4287872210 wait complete. 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 10 monclient: build_initial_monmap 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 10 monclient: monmap: 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:46:50.496 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 1 librados: starting msgr at 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 1 librados: starting objecter 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 1 Processor -- start 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 1 -- start start 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 1 librados: setting wanted keys 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 1 librados: calling monclient init 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.493+0000 7f88ab231640 10 monclient: init 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 10 monclient(hunting): picked mon.noname-c con 0x7f88a40b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 10 monclient(hunting): picked mon.noname-b con 0x7f88a40b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 10 monclient(hunting): picked mon.noname-a con 0x7f88a414be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a4138010 con 0x7f88a414be90 2024-03-25T19:46:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a41382f0 con 0x7f88a40b4d60 2024-03-25T19:46:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a4136d90 con 0x7f88a40b1730 2024-03-25T19:46:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.494+0000 7f88ab231640 10 monclient(hunting): authenticate will time out at 5237.622675s 2024-03-25T19:46:50.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.502+0000 7f88a97ce640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f88a414be90 0x7f88a41427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:53248/0 (socket says 172.21.15.18:53248) 2024-03-25T19:46:50.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.502+0000 7f88a97ce640 1 -- 172.21.15.18:0/1505204177 learned_addr learned my addr 172.21.15.18:0/1505204177 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:50.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.502+0000 7f88a17fa640 10 client.?.objecter ms_handle_connect 0x7f88a414be90 2024-03-25T19:46:50.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.503+0000 7f88a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:50.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.508+0000 7f88a17fa640 10 client.?.objecter ms_handle_connect 0x7f88a40b4d60 2024-03-25T19:46:50.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.508+0000 7f88a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:50.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.513+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772940392 0 0) 0x7f88a41382f0 con 0x7f88a40b4d60 2024-03-25T19:46:50.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.513+0000 7f88a17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.513+0000 7f88a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.513+0000 7f88a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.513+0000 7f88a17fa640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:46:50.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.513+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8888004eb0 con 0x7f88a40b4d60 2024-03-25T19:46:50.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.523+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1463119597 0 0) 0x7f8888004eb0 con 0x7f88a40b4d60 2024-03-25T19:46:50.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.523+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88a41382f0 con 0x7f88a40b4d60 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.523+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8890001930 con 0x7f88a40b4d60 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.523+0000 7f88a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 monclient(hunting): dump: 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:50.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:50.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 client.?.objecter ms_handle_connect 0x7f88a40b1730 2024-03-25T19:46:50.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:50.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303097815 0 0) 0x7f88a4138010 con 0x7f88a414be90 2024-03-25T19:46:50.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:46:50.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.524+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8888004eb0 con 0x7f88a414be90 2024-03-25T19:46:50.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.529+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863208109 0 0) 0x7f88a4136d90 con 0x7f88a40b1730 2024-03-25T19:46:50.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.529+0000 7f88a17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.529+0000 7f88a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.529+0000 7f88a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.529+0000 7f88a17fa640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:46:50.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.529+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a4138010 con 0x7f88a40b1730 2024-03-25T19:46:50.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 993072144 0 0) 0x7f88a41382f0 con 0x7f88a40b4d60 2024-03-25T19:46:50.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 >> v1:172.21.15.18:6790/0 conn(0x7f88a40b1730 legacy=0x7f88a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 >> v1:172.21.15.18:6789/0 conn(0x7f88a414be90 legacy=0x7f88a41427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 1 monclient: found mon.b 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4144010 con 0x7f88a40b4d60 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 10 monclient: _renew_subs 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:46:50.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a41372b0 con 0x7f88a40b4d60 2024-03-25T19:46:50.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:50.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 10 monclient: _renew_subs 2024-03-25T19:46:50.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:46:50.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a41369c0 con 0x7f88a40b4d60 2024-03-25T19:46:50.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.537+0000 7f88ab231640 1 librados: init done 2024-03-25T19:46:50.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.538+0000 7f88ab231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:50.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.538+0000 7f88ab231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:50.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.538+0000 7f88ab231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:46:50.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.538+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a4136f20 con 0x7f88a40b4d60 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.830+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88900047e0 con 0x7f88a40b4d60 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.830+0000 7f88a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.830+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8890004bf0 con 0x7f88a40b4d60 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.830+0000 7f88a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.831+0000 7f88a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.831+0000 7f88a17fa640 10 monclient: dump: 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:50.831 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.830+0000 7f88a2ffd640 4 set_mon_vals no callback set 2024-03-25T19:46:50.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.831+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f8890004e10 con 0x7f88a40b4d60 2024-03-25T19:46:50.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.832+0000 7f88a8fcd640 10 monclient: get_auth_request con 0x7f888806de10 auth_method 0 2024-03-25T19:46:50.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.832+0000 7f88a17fa640 10 client.4465.objecter ms_handle_connect 0x7f888806de10 2024-03-25T19:46:50.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.836+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f8890054810 con 0x7f88a40b4d60 2024-03-25T19:46:50.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.836+0000 7f88a17fa640 10 client.4465.objecter ms_dispatch 0x7f88a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:50.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.836+0000 7f88a17fa640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:50.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.836+0000 7f88a17fa640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:50.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.836+0000 7f88a17fa640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T19:46:50.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.837+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8890050320 con 0x7f88a40b4d60 2024-03-25T19:46:50.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.837+0000 7f88a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:50.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:50.837+0000 7f88a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:51.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.069+0000 7f88ab231640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.069+0000 7f88ab231640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.069+0000 7f88ab231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:46:51.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.069+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88a41441a0 con 0x7f88a40b4d60 2024-03-25T19:46:51.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.090+0000 7f88a17fa640 1 -- 172.21.15.18:0/1505204177 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f8890052250 con 0x7f88a40b4d60 2024-03-25T19:46:51.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.090+0000 7f88a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.090+0000 7f88a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:51.090 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:46:51.090 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.091+0000 7f88ab231640 10 librados: watch_flush enter 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.091+0000 7f88ab231640 10 librados: watch_flush exit 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.091+0000 7f88ab231640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.091+0000 7f88ab231640 10 client.4465.objecter successfully canceled tick 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 >> v1:172.21.15.18:6800/1520000573 conn(0x7f888806de10 legacy=0x7f88880702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 10 monclient: shutdown 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 >> v1:172.21.15.53:6789/0 conn(0x7f88a40b4d60 legacy=0x7f88a4143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 shutdown_connections 2024-03-25T19:46:51.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 >> 172.21.15.18:0/1505204177 conn(0x7f88a40617a0 msgr2=0x7f88a40b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:51.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 shutdown_connections 2024-03-25T19:46:51.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 1 -- 172.21.15.18:0/1505204177 wait complete. 2024-03-25T19:46:51.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:51.092+0000 7f88ab231640 1 librados: shutdown 2024-03-25T19:46:51.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:57.911 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:58.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.020+0000 7feacdf72640 10 monclient: get_monmap_and_config 2024-03-25T19:46:58.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.020+0000 7feacdf72640 10 monclient: build_initial_monmap 2024-03-25T19:46:58.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.021+0000 7feacdf72640 10 monclient: monmap: 2024-03-25T19:46:58.021 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:58.021 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:58.021 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:46:58.021817+0000 2024-03-25T19:46:58.021 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:46:58.021817+0000 2024-03-25T19:46:58.022 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:58.022 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:58.022 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:58.022 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:46:58.022 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:46:58.022 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:58.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.021+0000 7feacdf72640 1 Processor -- start 2024-03-25T19:46:58.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.021+0000 7feacdf72640 1 -- start start 2024-03-25T19:46:58.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.021+0000 7feacdf72640 10 monclient: init 2024-03-25T19:46:58.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:58.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 10 monclient(hunting): picked mon.noname-b con 0x7feac80b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 10 monclient(hunting): picked mon.noname-c con 0x7feac80b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 10 monclient(hunting): picked mon.noname-a con 0x7feac814be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80af940 con 0x7feac814be90 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80b0a70 con 0x7feac80b1730 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80b0d50 con 0x7feac80b4d60 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feacdf72640 10 monclient(hunting): authenticate will time out at 5245.149122s 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.022+0000 7feac77fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7feac80b4d60 0x7feac813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46484/0 (socket says 172.21.15.18:46484) 2024-03-25T19:46:58.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.023+0000 7feac77fe640 1 -- 172.21.15.18:0/349545151 learned_addr learned my addr 172.21.15.18:0/349545151 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:58.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.099+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752008469 0 0) 0x7feac80af940 con 0x7feac814be90 2024-03-25T19:46:58.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.099+0000 7feac67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.099+0000 7feac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.099+0000 7feac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.099+0000 7feac67fc640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:46:58.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.099+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0003520 con 0x7feac814be90 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3625166953 0 0) 0x7feab0003520 con 0x7feac814be90 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 >> v1:172.21.15.18:6790/0 conn(0x7feac80b4d60 legacy=0x7feac813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 >> v1:172.21.15.53:6789/0 conn(0x7feac80b1730 legacy=0x7feac80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 1 monclient: found mon.noname-a 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac80b1030 con 0x7feac814be90 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7feac0001a10 con 0x7feac814be90 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feac67fc640 10 monclient: dump: 2024-03-25T19:46:58.116 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feacdf72640 5 monclient: authenticate success, global_id 4500 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.115+0000 7feacdf72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:58.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.125+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feac0001b70 con 0x7feac814be90 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.125+0000 7feac67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.125+0000 7feac67fc640 1 -- 172.21.15.18:0/349545151 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7feac0001f80 con 0x7feac814be90 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.125+0000 7feacdf72640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.125+0000 7feacdf72640 4 set_mon_vals no callback set 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.125+0000 7feac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.125+0000 7feac67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.125+0000 7feac67fc640 10 monclient: dump: 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:58.125 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:58.126 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:58.126 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.126 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:58.126 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:58.126 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:58.126 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:58.126 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:58.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.126+0000 7feacdf72640 10 monclient: shutdown 2024-03-25T19:46:58.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.126+0000 7feacdf72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:58.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.126+0000 7feacdf72640 1 -- 172.21.15.18:0/349545151 >> v1:172.21.15.18:6789/0 conn(0x7feac814be90 legacy=0x7feac814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.126+0000 7feacdf72640 1 -- 172.21.15.18:0/349545151 shutdown_connections 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.126+0000 7feacdf72640 1 -- 172.21.15.18:0/349545151 >> 172.21.15.18:0/349545151 conn(0x7feac80617a0 msgr2=0x7feac8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.126+0000 7feacdf72640 1 -- 172.21.15.18:0/349545151 shutdown_connections 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.126+0000 7feacdf72640 1 -- 172.21.15.18:0/349545151 wait complete. 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 10 monclient: build_initial_monmap 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 10 monclient: monmap: 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 1 librados: starting msgr at 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 1 librados: starting objecter 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 1 Processor -- start 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 1 -- start start 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 1 librados: setting wanted keys 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 1 librados: calling monclient init 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.127+0000 7feacdf72640 10 monclient: init 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:58.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:58.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 10 monclient(hunting): picked mon.noname-c con 0x7feac80b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:46:58.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 10 monclient(hunting): picked mon.noname-b con 0x7feac80b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:46:58.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 10 monclient(hunting): picked mon.noname-a con 0x7feac814be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:46:58.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac8138150 con 0x7feac814be90 2024-03-25T19:46:58.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac8137c70 con 0x7feac80b4d60 2024-03-25T19:46:58.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac8137e00 con 0x7feac80b1730 2024-03-25T19:46:58.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:58.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.128+0000 7feacdf72640 10 monclient(hunting): authenticate will time out at 5245.255221s 2024-03-25T19:46:58.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.146+0000 7feac7fff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7feac814be90 0x7feac81d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47402/0 (socket says 172.21.15.18:47402) 2024-03-25T19:46:58.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.146+0000 7feac7fff640 1 -- 172.21.15.18:0/3623795384 learned_addr learned my addr 172.21.15.18:0/3623795384 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:46:58.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.147+0000 7feaaffff640 10 client.?.objecter ms_handle_connect 0x7feac814be90 2024-03-25T19:46:58.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.147+0000 7feaaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.157+0000 7feaaffff640 10 client.?.objecter ms_handle_connect 0x7feac80b4d60 2024-03-25T19:46:58.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.157+0000 7feaaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.162+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506659392 0 0) 0x7feac8138150 con 0x7feac814be90 2024-03-25T19:46:58.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.162+0000 7feaaffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.162+0000 7feaaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.162+0000 7feaaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.162+0000 7feaaffff640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:46:58.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.162+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab4004eb0 con 0x7feac814be90 2024-03-25T19:46:58.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.169+0000 7feaaffff640 10 client.?.objecter ms_handle_connect 0x7feac80b1730 2024-03-25T19:46:58.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.169+0000 7feaaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.189+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2027354292 0 0) 0x7feab4004eb0 con 0x7feac814be90 2024-03-25T19:46:58.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.189+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feac8138150 con 0x7feac814be90 2024-03-25T19:46:58.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.189+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7feac0001fc0 con 0x7feac814be90 2024-03-25T19:46:58.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.189+0000 7feaaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.189+0000 7feaaffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:58.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.189+0000 7feaaffff640 10 monclient(hunting): dump: 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:58.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.189+0000 7feaaffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:58.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2199780794 0 0) 0x7feac8138150 con 0x7feac814be90 2024-03-25T19:46:58.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 >> v1:172.21.15.18:6790/0 conn(0x7feac80b1730 legacy=0x7feac8131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 >> v1:172.21.15.53:6789/0 conn(0x7feac80b4d60 legacy=0x7feac81d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:58.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 1 monclient: found mon.a 2024-03-25T19:46:58.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:58.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac81386a0 con 0x7feac814be90 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 10 monclient: _finish_auth 0 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feaaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 5 monclient: authenticate success, global_id 4503 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 10 monclient: _renew_subs 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac8144e00 con 0x7feac814be90 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 10 monclient: _renew_subs 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feac8132690 con 0x7feac814be90 2024-03-25T19:46:58.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.260+0000 7feacdf72640 1 librados: init done 2024-03-25T19:46:58.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.261+0000 7feacdf72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:58.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.261+0000 7feacdf72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:58.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.261+0000 7feacdf72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:58.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.261+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac81451a0 con 0x7feac814be90 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.571+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feac0003c20 con 0x7feac814be90 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.572+0000 7feaaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.572+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7feac0004030 con 0x7feac814be90 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.572+0000 7feaaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.572+0000 7feaaffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.572+0000 7feaaffff640 10 monclient: dump: 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:58.572 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.572+0000 7feac5ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.572+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7feac0004250 con 0x7feac814be90 2024-03-25T19:46:58.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.573+0000 7feac77fe640 10 monclient: get_auth_request con 0x7feab406dc70 auth_method 0 2024-03-25T19:46:58.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.573+0000 7feaaffff640 10 client.4503.objecter ms_handle_connect 0x7feab406dc70 2024-03-25T19:46:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.583+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7feac0054bb0 con 0x7feac814be90 2024-03-25T19:46:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.583+0000 7feaaffff640 10 client.4503.objecter ms_dispatch 0x7feac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.583+0000 7feaaffff640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.583+0000 7feaaffff640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.583+0000 7feaaffff640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T19:46:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.583+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feac00506c0 con 0x7feac814be90 2024-03-25T19:46:58.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.583+0000 7feaaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:58.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.583+0000 7feaaffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:58.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.826+0000 7feacdf72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:58.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.827+0000 7feacdf72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:58.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.827+0000 7feacdf72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:46:58.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.827+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feac80627b0 con 0x7feac814be90 2024-03-25T19:46:58.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.871+0000 7feaaffff640 1 -- 172.21.15.18:0/3623795384 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7feac00525f0 con 0x7feac814be90 2024-03-25T19:46:58.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.871+0000 7feaaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:58.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.871+0000 7feaaffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:58.871 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:46:58.871 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:58.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 10 librados: watch_flush enter 2024-03-25T19:46:58.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 10 librados: watch_flush exit 2024-03-25T19:46:58.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T19:46:58.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 10 client.4503.objecter successfully canceled tick 2024-03-25T19:46:58.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 >> v1:172.21.15.18:6800/1520000573 conn(0x7feab406dc70 legacy=0x7feab4070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 10 monclient: shutdown 2024-03-25T19:46:58.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:58.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 >> v1:172.21.15.18:6789/0 conn(0x7feac814be90 legacy=0x7feac81d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 shutdown_connections 2024-03-25T19:46:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 >> 172.21.15.18:0/3623795384 conn(0x7feac80617a0 msgr2=0x7feac80b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 shutdown_connections 2024-03-25T19:46:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 1 -- 172.21.15.18:0/3623795384 wait complete. 2024-03-25T19:46:58.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:46:58.873+0000 7feacdf72640 1 librados: shutdown 2024-03-25T19:46:58.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:05.692 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.808+0000 7f3f82d58640 10 monclient: get_monmap_and_config 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.808+0000 7f3f82d58640 10 monclient: build_initial_monmap 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.808+0000 7f3f82d58640 10 monclient: monmap: 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:47:05.809158+0000 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:47:05.809158+0000 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:05.808 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:05.809 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:47:05.809 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:47:05.809 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 1 Processor -- start 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 1 -- start start 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 10 monclient: init 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:05.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:05.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c142520 con 0x7f3f7c1430c0 2024-03-25T19:47:05.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.809+0000 7f3f82d58640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0b28a0 con 0x7f3f7c13f710 2024-03-25T19:47:05.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.810+0000 7f3f82d58640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c0b2a30 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.810+0000 7f3f82d58640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:05.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.810+0000 7f3f82d58640 10 monclient(hunting): authenticate will time out at 5252.936755s 2024-03-25T19:47:05.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.818+0000 7f3f80af4640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3f7c0a98f0 0x7f3f7c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46688/0 (socket says 172.21.15.18:46688) 2024-03-25T19:47:05.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.818+0000 7f3f80af4640 1 -- 172.21.15.18:0/63739042 learned_addr learned my addr 172.21.15.18:0/63739042 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:05.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.838+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648334976 0 0) 0x7f3f7c0b28a0 con 0x7f3f7c13f710 2024-03-25T19:47:05.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.838+0000 7f3f7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.838+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.838+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.838+0000 7f3f7b7fe640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:47:05.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.838+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c003520 con 0x7f3f7c13f710 2024-03-25T19:47:05.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.839+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846209859 0 0) 0x7f3f7c0b2a30 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.839+0000 7f3f7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.839+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.839+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.839+0000 7f3f7b7fe640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.839+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c0b28a0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.847+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360240086 0 0) 0x7f3f7c142520 con 0x7f3f7c1430c0 2024-03-25T19:47:05.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.847+0000 7f3f7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.847+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.847+0000 7f3f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.847+0000 7f3f7b7fe640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:47:05.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.847+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c0b2a30 con 0x7f3f7c1430c0 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 229384717 0 0) 0x7f3f7c0b28a0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 >> v1:172.21.15.53:6789/0 conn(0x7f3f7c13f710 legacy=0x7f3f7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 >> v1:172.21.15.18:6789/0 conn(0x7f3f7c1430c0 legacy=0x7f3f7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c0b44c0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3f70001670 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.848+0000 7f3f82d58640 5 monclient: authenticate success, global_id 4472 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.849+0000 7f3f82d58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.849+0000 7f3f7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.849+0000 7f3f7b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.849+0000 7f3f7b7fe640 10 monclient: dump: 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:05.850 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.851 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.851 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.851 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:05.851 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:05.851 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:05.851 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:05.851 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:05.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.849+0000 7f3f82d58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f700017d0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f7b7fe640 1 -- 172.21.15.18:0/63739042 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3f70001be0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f7b7fe640 10 monclient: dump: 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.856 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:05.857 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:05.857 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:05.857 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:05.857 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:05.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f82d58640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:05.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f82d58640 4 set_mon_vals no callback set 2024-03-25T19:47:05.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f82d58640 10 monclient: shutdown 2024-03-25T19:47:05.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f82d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:05.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.856+0000 7f3f82d58640 1 -- 172.21.15.18:0/63739042 >> v1:172.21.15.18:6790/0 conn(0x7f3f7c0a98f0 legacy=0x7f3f7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.857+0000 7f3f82d58640 1 -- 172.21.15.18:0/63739042 shutdown_connections 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.857+0000 7f3f82d58640 1 -- 172.21.15.18:0/63739042 >> 172.21.15.18:0/63739042 conn(0x7f3f7c0b7ea0 msgr2=0x7f3f7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.857+0000 7f3f82d58640 1 -- 172.21.15.18:0/63739042 shutdown_connections 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.857+0000 7f3f82d58640 1 -- 172.21.15.18:0/63739042 wait complete. 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.857+0000 7f3f82d58640 10 monclient: build_initial_monmap 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.857+0000 7f3f82d58640 10 monclient: monmap: 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:47:05.860 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.858+0000 7f3f82d58640 1 librados: starting msgr at 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.858+0000 7f3f82d58640 1 librados: starting objecter 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.858+0000 7f3f82d58640 1 Processor -- start 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.858+0000 7f3f82d58640 1 -- start start 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.858+0000 7f3f82d58640 1 librados: setting wanted keys 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.858+0000 7f3f82d58640 1 librados: calling monclient init 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.858+0000 7f3f82d58640 10 monclient: init 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 10 monclient(hunting): picked mon.noname-b con 0x7f3f7c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c136e60 con 0x7f3f7c1430c0 2024-03-25T19:47:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c1381a0 con 0x7f3f7c13f710 2024-03-25T19:47:05.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c1370b0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:05.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.859+0000 7f3f82d58640 10 monclient(hunting): authenticate will time out at 5252.985954s 2024-03-25T19:47:05.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.870+0000 7f3f80af4640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3f7c0a98f0 0x7f3f7c137a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46718/0 (socket says 172.21.15.18:46718) 2024-03-25T19:47:05.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.870+0000 7f3f80af4640 1 -- 172.21.15.18:0/1312557970 learned_addr learned my addr 172.21.15.18:0/1312557970 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:05.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.875+0000 7f3f797fa640 10 client.?.objecter ms_handle_connect 0x7f3f7c0a98f0 2024-03-25T19:47:05.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.875+0000 7f3f797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:05.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.884+0000 7f3f797fa640 10 client.?.objecter ms_handle_connect 0x7f3f7c13f710 2024-03-25T19:47:05.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.884+0000 7f3f797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:05.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.885+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961022388 0 0) 0x7f3f7c1370b0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.885+0000 7f3f797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.885+0000 7f3f797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.885+0000 7f3f797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.885+0000 7f3f797fa640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:47:05.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.885+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f60004eb0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.891+0000 7f3f797fa640 10 client.?.objecter ms_handle_connect 0x7f3f7c1430c0 2024-03-25T19:47:05.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.891+0000 7f3f797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:05.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.899+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708071294 0 0) 0x7f3f7c1381a0 con 0x7f3f7c13f710 2024-03-25T19:47:05.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.899+0000 7f3f797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.899+0000 7f3f797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.899+0000 7f3f797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.899+0000 7f3f797fa640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:47:05.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.899+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c1370b0 con 0x7f3f7c13f710 2024-03-25T19:47:05.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236266674 0 0) 0x7f3f7c136e60 con 0x7f3f7c1430c0 2024-03-25T19:47:05.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c1381a0 con 0x7f3f7c1430c0 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762489887 0 0) 0x7f3f60004eb0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f7c136e60 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3f70001c20 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 10 monclient(hunting): dump: 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:05.904 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:05.905 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:05.905 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:05.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.903+0000 7f3f797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.913+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4166255069 0 0) 0x7f3f7c1370b0 con 0x7f3f7c13f710 2024-03-25T19:47:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.913+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f60004eb0 con 0x7f3f7c13f710 2024-03-25T19:47:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.913+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3f68001950 con 0x7f3f7c13f710 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.913+0000 7f3f797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.914+0000 7f3f797fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.914+0000 7f3f797fa640 10 monclient(hunting): dump: 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:05.914 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.925+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106474421 0 0) 0x7f3f7c1381a0 con 0x7f3f7c1430c0 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.925+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f7c1370b0 con 0x7f3f7c1430c0 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.925+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3f74001bd0 con 0x7f3f7c1430c0 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.925+0000 7f3f797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.925+0000 7f3f797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.925+0000 7f3f797fa640 10 monclient(hunting): dump: 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:05.925 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.926 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:05.926 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:05.926 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:05.926 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:05.926 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:05.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4266841720 0 0) 0x7f3f7c136e60 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 >> v1:172.21.15.53:6789/0 conn(0x7f3f7c13f710 legacy=0x7f3f7c13a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 >> v1:172.21.15.18:6789/0 conn(0x7f3f7c1430c0 legacy=0x7f3f7c13ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 10 monclient: _finish_hunting 0 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 1 monclient: found mon.c 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c137390 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 10 monclient: _finish_auth 0 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 10 monclient: _check_auth_tickets 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:05.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 5 monclient: authenticate success, global_id 4475 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 10 monclient: _renew_subs 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c1376d0 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 10 monclient: _renew_subs 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c135530 con 0x7f3f7c0a98f0 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.926+0000 7f3f82d58640 1 librados: init done 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.927+0000 7f3f82d58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.927+0000 7f3f82d58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.927+0000 7f3f82d58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:05.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:05.927+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c0b3ef0 con 0x7f3f7c0a98f0 2024-03-25T19:47:06.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.151+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f70004a90 con 0x7f3f7c0a98f0 2024-03-25T19:47:06.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.151+0000 7f3f797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:06.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.151+0000 7f3f7affd640 4 set_mon_vals no callback set 2024-03-25T19:47:06.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.237+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f3f70004de0 con 0x7f3f7c0a98f0 2024-03-25T19:47:06.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.237+0000 7f3f797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.237+0000 7f3f797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.237+0000 7f3f797fa640 10 monclient: dump: 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:06.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.237+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f3f70004fc0 con 0x7f3f7c0a98f0 2024-03-25T19:47:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.238+0000 7f3f7bfff640 10 monclient: get_auth_request con 0x7f3f6006e0f0 auth_method 0 2024-03-25T19:47:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.238+0000 7f3f797fa640 10 client.4475.objecter ms_handle_connect 0x7f3f6006e0f0 2024-03-25T19:47:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.245+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f3f70054a20 con 0x7f3f7c0a98f0 2024-03-25T19:47:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.245+0000 7f3f797fa640 10 client.4475.objecter ms_dispatch 0x7f3f7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.245+0000 7f3f797fa640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.245+0000 7f3f797fa640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.245+0000 7f3f797fa640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-25T19:47:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.245+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f70050530 con 0x7f3f7c0a98f0 2024-03-25T19:47:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.245+0000 7f3f797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.245+0000 7f3f797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:06.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.463+0000 7f3f82d58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:06.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.463+0000 7f3f82d58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:06.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.463+0000 7f3f82d58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:06.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.463+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f7c1357e0 con 0x7f3f7c0a98f0 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.587+0000 7f3f797fa640 1 -- 172.21.15.18:0/1312557970 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f3f70052460 con 0x7f3f7c0a98f0 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.587+0000 7f3f797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.587+0000 7f3f797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:06.587 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:47:06.588 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 10 librados: watch_flush enter 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 10 librados: watch_flush exit 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 10 client.4475.objecter successfully canceled tick 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 >> v1:172.21.15.18:6800/1520000573 conn(0x7f3f6006e0f0 legacy=0x7f3f600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 10 monclient: shutdown 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 >> v1:172.21.15.18:6790/0 conn(0x7f3f7c0a98f0 legacy=0x7f3f7c137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 shutdown_connections 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 >> 172.21.15.18:0/1312557970 conn(0x7f3f7c0b7ea0 msgr2=0x7f3f7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 shutdown_connections 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.589+0000 7f3f82d58640 1 -- 172.21.15.18:0/1312557970 wait complete. 2024-03-25T19:47:06.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:06.590+0000 7f3f82d58640 1 librados: shutdown 2024-03-25T19:47:06.602 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:13.118 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:40.648 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.123 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:40.653 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.149 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:40.679 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.176 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:06:40.672 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.178 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:40.709 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.185 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:40.716 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.187 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:06:40.684 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.215 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:40.745 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.229 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:06:40.725 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:13.408 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.524+0000 7fa35b3e8640 10 monclient: get_monmap_and_config 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.524+0000 7fa35b3e8640 10 monclient: build_initial_monmap 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.524+0000 7fa35b3e8640 10 monclient: monmap: 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:47:13.525794+0000 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:47:13.525794+0000 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:47:13.525 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:13.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.525+0000 7fa35b3e8640 1 Processor -- start 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 1 -- start start 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 10 monclient: init 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 10 monclient(hunting): picked mon.noname-c con 0x7fa3540a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 10 monclient(hunting): picked mon.noname-b con 0x7fa35413f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 10 monclient(hunting): picked mon.noname-a con 0x7fa3541430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa354142520 con 0x7fa3541430c0 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540b28a0 con 0x7fa3540a98f0 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540b2a30 con 0x7fa35413f710 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:13.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.526+0000 7fa35b3e8640 10 monclient(hunting): authenticate will time out at 5260.653576s 2024-03-25T19:47:13.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.532+0000 7fa359184640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa3540a98f0 0x7fa3540a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:44872/0 (socket says 172.21.15.18:44872) 2024-03-25T19:47:13.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.532+0000 7fa359184640 1 -- 172.21.15.18:0/4043228725 learned_addr learned my addr 172.21.15.18:0/4043228725 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:13.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.549+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509367525 0 0) 0x7fa354142520 con 0x7fa3541430c0 2024-03-25T19:47:13.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.549+0000 7fa34bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.549+0000 7fa34bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.549+0000 7fa34bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.549+0000 7fa34bfff640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:47:13.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.549+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa340003520 con 0x7fa3541430c0 2024-03-25T19:47:13.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.551+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621192975 0 0) 0x7fa3540b28a0 con 0x7fa3540a98f0 2024-03-25T19:47:13.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.551+0000 7fa34bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.551+0000 7fa34bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.551+0000 7fa34bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.551+0000 7fa34bfff640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:47:13.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.551+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa354142520 con 0x7fa3540a98f0 2024-03-25T19:47:13.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1118380953 0 0) 0x7fa340003520 con 0x7fa3541430c0 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 >> v1:172.21.15.18:6790/0 conn(0x7fa35413f710 legacy=0x7fa3541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 >> v1:172.21.15.53:6789/0 conn(0x7fa3540a98f0 legacy=0x7fa3540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 1 monclient: found mon.noname-a 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540b44c0 con 0x7fa3541430c0 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.560+0000 7fa34bfff640 10 monclient: _finish_auth 0 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.561+0000 7fa34bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.561+0000 7fa34bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.561+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa34c001a10 con 0x7fa3541430c0 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.561+0000 7fa34bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.561+0000 7fa34bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.561+0000 7fa34bfff640 10 monclient: dump: 2024-03-25T19:47:13.561 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.561+0000 7fa35b3e8640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:47:13.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.561+0000 7fa35b3e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:13.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.568+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa34c001b70 con 0x7fa3541430c0 2024-03-25T19:47:13.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.568+0000 7fa34bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:13.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.568+0000 7fa34bfff640 1 -- 172.21.15.18:0/4043228725 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa34c001f80 con 0x7fa3541430c0 2024-03-25T19:47:13.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.568+0000 7fa35b3e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:13.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.568+0000 7fa35b3e8640 4 set_mon_vals no callback set 2024-03-25T19:47:13.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.568+0000 7fa34bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.568+0000 7fa34bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.568+0000 7fa34bfff640 10 monclient: dump: 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:13.569 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:13.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.569+0000 7fa35b3e8640 10 monclient: shutdown 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.569+0000 7fa35b3e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.569+0000 7fa35b3e8640 1 -- 172.21.15.18:0/4043228725 >> v1:172.21.15.18:6789/0 conn(0x7fa3541430c0 legacy=0x7fa35414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.569+0000 7fa35b3e8640 1 -- 172.21.15.18:0/4043228725 shutdown_connections 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.569+0000 7fa35b3e8640 1 -- 172.21.15.18:0/4043228725 >> 172.21.15.18:0/4043228725 conn(0x7fa3540b7ea0 msgr2=0x7fa3540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.569+0000 7fa35b3e8640 1 -- 172.21.15.18:0/4043228725 shutdown_connections 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.569+0000 7fa35b3e8640 1 -- 172.21.15.18:0/4043228725 wait complete. 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.570+0000 7fa35b3e8640 10 monclient: build_initial_monmap 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.570+0000 7fa35b3e8640 10 monclient: monmap: 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:47:13.571 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.570+0000 7fa35b3e8640 1 librados: starting msgr at 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.570+0000 7fa35b3e8640 1 librados: starting objecter 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 1 Processor -- start 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 1 -- start start 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 1 librados: setting wanted keys 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 1 librados: calling monclient init 2024-03-25T19:47:13.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 10 monclient: init 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 10 monclient(hunting): picked mon.noname-b con 0x7fa3540a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 10 monclient(hunting): picked mon.noname-a con 0x7fa35413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 10 monclient(hunting): picked mon.noname-c con 0x7fa3541430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa354139830 con 0x7fa35413f710 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.571+0000 7fa35b3e8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3541db7d0 con 0x7fa3540a98f0 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.572+0000 7fa35b3e8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3541db410 con 0x7fa3541430c0 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.572+0000 7fa35b3e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:13.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.572+0000 7fa35b3e8640 10 monclient(hunting): authenticate will time out at 5260.698761s 2024-03-25T19:47:13.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.578+0000 7fa359184640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa3540a98f0 0x7fa3541dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:44908/0 (socket says 172.21.15.18:44908) 2024-03-25T19:47:13.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.578+0000 7fa359184640 1 -- 172.21.15.18:0/274359440 learned_addr learned my addr 172.21.15.18:0/274359440 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:13.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.581+0000 7fa349ffb640 10 client.?.objecter ms_handle_connect 0x7fa3540a98f0 2024-03-25T19:47:13.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.581+0000 7fa349ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.587+0000 7fa349ffb640 10 client.?.objecter ms_handle_connect 0x7fa3541430c0 2024-03-25T19:47:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.587+0000 7fa349ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:13.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.603+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896911501 0 0) 0x7fa3541db7d0 con 0x7fa3540a98f0 2024-03-25T19:47:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.603+0000 7fa349ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.603+0000 7fa349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.603+0000 7fa349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.603+0000 7fa349ffb640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:47:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.603+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa338004eb0 con 0x7fa3540a98f0 2024-03-25T19:47:13.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.611+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994092414 0 0) 0x7fa3541db410 con 0x7fa3541430c0 2024-03-25T19:47:13.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.611+0000 7fa349ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.611+0000 7fa349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.611+0000 7fa349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.611+0000 7fa349ffb640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:47:13.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.611+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3541db7d0 con 0x7fa3541430c0 2024-03-25T19:47:13.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.616+0000 7fa349ffb640 10 client.?.objecter ms_handle_connect 0x7fa35413f710 2024-03-25T19:47:13.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.616+0000 7fa349ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.623+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 964295845 0 0) 0x7fa3541db7d0 con 0x7fa3541430c0 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.623+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3541db410 con 0x7fa3541430c0 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.623+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa34c001e80 con 0x7fa3541430c0 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.623+0000 7fa349ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.623+0000 7fa349ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.623+0000 7fa349ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:13.623 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:13.624 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.624 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:13.624 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:13.624 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:13.624 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:13.624 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:13.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.623+0000 7fa349ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:13.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.628+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422920271 0 0) 0x7fa354139830 con 0x7fa35413f710 2024-03-25T19:47:13.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.628+0000 7fa349ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.628+0000 7fa349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.628+0000 7fa349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.628+0000 7fa349ffb640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:47:13.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.628+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3541db7d0 con 0x7fa35413f710 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 85973700 0 0) 0x7fa3541db410 con 0x7fa3541430c0 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 >> v1:172.21.15.53:6789/0 conn(0x7fa3540a98f0 legacy=0x7fa3541dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 >> v1:172.21.15.18:6789/0 conn(0x7fa35413f710 legacy=0x7fa3540b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 1 monclient: found mon.c 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3541db5a0 con 0x7fa3541430c0 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 10 monclient: _finish_auth 0 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa349ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 5 monclient: authenticate success, global_id 4481 2024-03-25T19:47:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 10 monclient: _renew_subs 2024-03-25T19:47:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3540b1480 con 0x7fa3541430c0 2024-03-25T19:47:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 10 monclient: _renew_subs 2024-03-25T19:47:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3540b0b10 con 0x7fa3541430c0 2024-03-25T19:47:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.633+0000 7fa35b3e8640 1 librados: init done 2024-03-25T19:47:13.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.635+0000 7fa35b3e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:13.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.635+0000 7fa35b3e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:13.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.635+0000 7fa35b3e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:13.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.635+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3541db960 con 0x7fa3541430c0 2024-03-25T19:47:13.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.855+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa34c001fe0 con 0x7fa3541430c0 2024-03-25T19:47:13.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.855+0000 7fa349ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:13.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.855+0000 7fa34b7fe640 4 set_mon_vals no callback set 2024-03-25T19:47:13.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.936+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa34c003dc0 con 0x7fa3541430c0 2024-03-25T19:47:13.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.936+0000 7fa349ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.936+0000 7fa349ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.936+0000 7fa349ffb640 10 monclient: dump: 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:13.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.937+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fa34c019d90 con 0x7fa3541430c0 2024-03-25T19:47:13.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.938+0000 7fa359184640 10 monclient: get_auth_request con 0x7fa33806dd70 auth_method 0 2024-03-25T19:47:13.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.938+0000 7fa349ffb640 10 client.4481.objecter ms_handle_connect 0x7fa33806dd70 2024-03-25T19:47:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.945+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fa34c053940 con 0x7fa3541430c0 2024-03-25T19:47:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.945+0000 7fa349ffb640 10 client.4481.objecter ms_dispatch 0x7fa354000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.945+0000 7fa349ffb640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.945+0000 7fa349ffb640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.945+0000 7fa349ffb640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-25T19:47:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.945+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa34c04f450 con 0x7fa3541430c0 2024-03-25T19:47:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.945+0000 7fa349ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:13.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:13.945+0000 7fa349ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:14.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.186+0000 7fa35b3e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.186+0000 7fa35b3e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.186+0000 7fa35b3e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:14.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.186+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3540627b0 con 0x7fa3541430c0 2024-03-25T19:47:14.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.286+0000 7fa349ffb640 1 -- 172.21.15.18:0/274359440 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fa34c051380 con 0x7fa3541430c0 2024-03-25T19:47:14.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.286+0000 7fa349ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.286+0000 7fa349ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:14.287 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:47:14.287 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:14.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 10 librados: watch_flush enter 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 10 librados: watch_flush exit 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 10 client.4481.objecter successfully canceled tick 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 >> v1:172.21.15.18:6800/1520000573 conn(0x7fa33806dd70 legacy=0x7fa338070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 10 monclient: shutdown 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 >> v1:172.21.15.18:6790/0 conn(0x7fa3541430c0 legacy=0x7fa3541d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 shutdown_connections 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.289+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 >> 172.21.15.18:0/274359440 conn(0x7fa3540b7ea0 msgr2=0x7fa354074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.290+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 shutdown_connections 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.290+0000 7fa35b3e8640 1 -- 172.21.15.18:0/274359440 wait complete. 2024-03-25T19:47:14.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:14.290+0000 7fa35b3e8640 1 librados: shutdown 2024-03-25T19:47:14.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:21.107 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:21.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.220+0000 7faeb1d93640 10 monclient: get_monmap_and_config 2024-03-25T19:47:21.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.220+0000 7faeb1d93640 10 monclient: build_initial_monmap 2024-03-25T19:47:21.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.220+0000 7faeb1d93640 10 monclient: monmap: 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:47:21.221524+0000 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:47:21.221524+0000 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:47:21.221 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.221+0000 7faeb1d93640 1 Processor -- start 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.221+0000 7faeb1d93640 1 -- start start 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.221+0000 7faeb1d93640 10 monclient: init 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 10 monclient(hunting): picked mon.noname-c con 0x7faeac061890 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 10 monclient(hunting): picked mon.noname-b con 0x7faeac06e740 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 10 monclient(hunting): picked mon.noname-a con 0x7faeac0ae280 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac0ae830 con 0x7faeac0ae280 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac06f720 con 0x7faeac061890 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac0ada80 con 0x7faeac06e740 2024-03-25T19:47:21.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:21.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.222+0000 7faeb1d93640 10 monclient(hunting): authenticate will time out at 5268.349469s 2024-03-25T19:47:21.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.233+0000 7faeab7fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faeac061890 0x7faeac13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:50998/0 (socket says 172.21.15.18:50998) 2024-03-25T19:47:21.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.233+0000 7faeab7fe640 1 -- 172.21.15.18:0/1635693542 learned_addr learned my addr 172.21.15.18:0/1635693542 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:21.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.243+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211245384 0 0) 0x7faeac06f720 con 0x7faeac061890 2024-03-25T19:47:21.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.243+0000 7faeaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.243+0000 7faeaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.243+0000 7faeaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.243+0000 7faeaa7fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:47:21.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.243+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae94003520 con 0x7faeac061890 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.268+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518099106 0 0) 0x7faeac0ae830 con 0x7faeac0ae280 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.268+0000 7faeaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.268+0000 7faeaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.268+0000 7faeaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.268+0000 7faeaa7fc640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.269+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeac06f720 con 0x7faeac0ae280 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.269+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508211477 0 0) 0x7faeac0ada80 con 0x7faeac06e740 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.269+0000 7faeaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.269+0000 7faeaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.269+0000 7faeaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.269+0000 7faeaa7fc640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:47:21.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.269+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeac0ae830 con 0x7faeac06e740 2024-03-25T19:47:21.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.279+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168022157 0 0) 0x7fae94003520 con 0x7faeac061890 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.279+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 >> v1:172.21.15.18:6790/0 conn(0x7faeac06e740 legacy=0x7faeac145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 >> v1:172.21.15.18:6789/0 conn(0x7faeac0ae280 legacy=0x7faeac145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 1 monclient: found mon.noname-c 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac13fbd0 con 0x7faeac061890 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fae9c001690 con 0x7faeac061890 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeaa7fc640 10 monclient: dump: 2024-03-25T19:47:21.280 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeb1d93640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:47:21.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.280+0000 7faeb1d93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:21.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.287+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae9c0017f0 con 0x7faeac061890 2024-03-25T19:47:21.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.287+0000 7faeaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:21.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.287+0000 7faeaa7fc640 1 -- 172.21.15.18:0/1635693542 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fae9c001c00 con 0x7faeac061890 2024-03-25T19:47:21.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.287+0000 7faeaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.287+0000 7faeaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.287+0000 7faeaa7fc640 10 monclient: dump: 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.287+0000 7faeb1d93640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:21.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.287+0000 7faeb1d93640 4 set_mon_vals no callback set 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.288+0000 7faeb1d93640 10 monclient: shutdown 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.288+0000 7faeb1d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.288+0000 7faeb1d93640 1 -- 172.21.15.18:0/1635693542 >> v1:172.21.15.53:6789/0 conn(0x7faeac061890 legacy=0x7faeac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.288+0000 7faeb1d93640 1 -- 172.21.15.18:0/1635693542 shutdown_connections 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.288+0000 7faeb1d93640 1 -- 172.21.15.18:0/1635693542 >> 172.21.15.18:0/1635693542 conn(0x7faeac0b0ef0 msgr2=0x7faeac0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.288+0000 7faeb1d93640 1 -- 172.21.15.18:0/1635693542 shutdown_connections 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.288+0000 7faeb1d93640 1 -- 172.21.15.18:0/1635693542 wait complete. 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.289+0000 7faeb1d93640 10 monclient: build_initial_monmap 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.289+0000 7faeb1d93640 10 monclient: monmap: 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:47:21.291 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.289+0000 7faeb1d93640 1 librados: starting msgr at 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.289+0000 7faeb1d93640 1 librados: starting objecter 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 1 Processor -- start 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 1 -- start start 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 1 librados: setting wanted keys 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 1 librados: calling monclient init 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 10 monclient: init 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:21.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 10 monclient(hunting): picked mon.noname-c con 0x7faeac061890 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:21.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 10 monclient(hunting): picked mon.noname-a con 0x7faeac0ae280 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:21.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.290+0000 7faeb1d93640 10 monclient(hunting): picked mon.noname-b con 0x7faeac142410 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:21.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.291+0000 7faeb1d93640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac1dbe00 con 0x7faeac0ae280 2024-03-25T19:47:21.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.291+0000 7faeb1d93640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac135a80 con 0x7faeac142410 2024-03-25T19:47:21.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.291+0000 7faeb1d93640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeac135d60 con 0x7faeac061890 2024-03-25T19:47:21.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.291+0000 7faeb1d93640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:21.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.291+0000 7faeb1d93640 10 monclient(hunting): authenticate will time out at 5268.417861s 2024-03-25T19:47:21.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.298+0000 7faeaaffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7faeac0ae280 0x7faeac1dafe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41422/0 (socket says 172.21.15.18:41422) 2024-03-25T19:47:21.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.298+0000 7faeaaffd640 1 -- 172.21.15.18:0/2704094004 learned_addr learned my addr 172.21.15.18:0/2704094004 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:21.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.299+0000 7fae93fff640 10 client.?.objecter ms_handle_connect 0x7faeac0ae280 2024-03-25T19:47:21.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.299+0000 7fae93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.309+0000 7fae93fff640 10 client.?.objecter ms_handle_connect 0x7faeac142410 2024-03-25T19:47:21.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.309+0000 7fae93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.313+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834806887 0 0) 0x7faeac1dbe00 con 0x7faeac0ae280 2024-03-25T19:47:21.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.313+0000 7fae93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.313+0000 7fae93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.313+0000 7fae93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.313+0000 7fae93fff640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:47:21.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.313+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae98004eb0 con 0x7faeac0ae280 2024-03-25T19:47:21.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.315+0000 7fae93fff640 10 client.?.objecter ms_handle_connect 0x7faeac061890 2024-03-25T19:47:21.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.315+0000 7fae93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.320+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062193872 0 0) 0x7faeac135a80 con 0x7faeac142410 2024-03-25T19:47:21.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.320+0000 7fae93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.320+0000 7fae93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.320+0000 7fae93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.320+0000 7fae93fff640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:47:21.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.320+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeac1dbe00 con 0x7faeac142410 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.324+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1025349297 0 0) 0x7fae98004eb0 con 0x7faeac0ae280 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.324+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faeac135a80 con 0x7faeac0ae280 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.324+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faea0001a10 con 0x7faeac0ae280 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.324+0000 7fae93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.324+0000 7fae93fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.324+0000 7fae93fff640 10 monclient(hunting): dump: 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:21.324 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:21.325 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:21.325 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:21.325 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:21.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.324+0000 7fae93fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:21.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.326+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233590595 0 0) 0x7faeac135d60 con 0x7faeac061890 2024-03-25T19:47:21.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.326+0000 7fae93fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.326+0000 7fae93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.326+0000 7fae93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.326+0000 7fae93fff640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:47:21.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.326+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae98004eb0 con 0x7faeac061890 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.336+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3011568840 0 0) 0x7faeac135a80 con 0x7faeac0ae280 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 >> v1:172.21.15.18:6790/0 conn(0x7faeac061890 legacy=0x7faeac141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 >> v1:172.21.15.53:6789/0 conn(0x7faeac142410 legacy=0x7faeac1db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 1 monclient: found mon.a 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac134820 con 0x7faeac0ae280 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:21.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7fae93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 5 monclient: authenticate success, global_id 4521 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 10 monclient: _renew_subs 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeac143020 con 0x7faeac0ae280 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 10 monclient: _renew_subs 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faeac1361f0 con 0x7faeac0ae280 2024-03-25T19:47:21.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.337+0000 7faeb1d93640 1 librados: init done 2024-03-25T19:47:21.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.338+0000 7faeb1d93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:21.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.338+0000 7faeb1d93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:21.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.338+0000 7faeb1d93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:47:21.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.338+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeac1410b0 con 0x7faeac0ae280 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.659+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faea0001c70 con 0x7faeac0ae280 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.659+0000 7fae93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.659+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faea0003a60 con 0x7faeac0ae280 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.659+0000 7fae93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.659+0000 7fae93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.659+0000 7fae93fff640 10 monclient: dump: 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.659+0000 7faea9ffb640 4 set_mon_vals no callback set 2024-03-25T19:47:21.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.660+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7faea0003c80 con 0x7faeac0ae280 2024-03-25T19:47:21.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.660+0000 7faeab7fe640 10 monclient: get_auth_request con 0x7fae9806de00 auth_method 0 2024-03-25T19:47:21.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.661+0000 7fae93fff640 10 client.4521.objecter ms_handle_connect 0x7fae9806de00 2024-03-25T19:47:21.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.668+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7faea0054860 con 0x7faeac0ae280 2024-03-25T19:47:21.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.668+0000 7fae93fff640 10 client.4521.objecter ms_dispatch 0x7faeac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:21.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.668+0000 7fae93fff640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:21.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.668+0000 7fae93fff640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:21.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.668+0000 7fae93fff640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T19:47:21.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.668+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faea0050370 con 0x7faeac0ae280 2024-03-25T19:47:21.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.668+0000 7fae93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:21.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.668+0000 7fae93fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:21.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.901+0000 7faeb1d93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:21.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.901+0000 7faeb1d93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:21.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.901+0000 7faeb1d93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:47:21.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.901+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeac134d40 con 0x7faeac0ae280 2024-03-25T19:47:21.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.934+0000 7fae93fff640 1 -- 172.21.15.18:0/2704094004 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7faea00522a0 con 0x7faeac0ae280 2024-03-25T19:47:21.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.934+0000 7fae93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:21.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.934+0000 7fae93fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:21.935 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:47:21.935 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:21.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.936+0000 7faeb1d93640 10 librados: watch_flush enter 2024-03-25T19:47:21.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.936+0000 7faeb1d93640 10 librados: watch_flush exit 2024-03-25T19:47:21.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.936+0000 7faeb1d93640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T19:47:21.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 10 client.4521.objecter successfully canceled tick 2024-03-25T19:47:21.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 >> v1:172.21.15.18:6800/1520000573 conn(0x7fae9806de00 legacy=0x7fae980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 10 monclient: shutdown 2024-03-25T19:47:21.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:21.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 >> v1:172.21.15.18:6789/0 conn(0x7faeac0ae280 legacy=0x7faeac1dafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 shutdown_connections 2024-03-25T19:47:21.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 >> 172.21.15.18:0/2704094004 conn(0x7faeac0b0ef0 msgr2=0x7faeac0b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:21.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 shutdown_connections 2024-03-25T19:47:21.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 1 -- 172.21.15.18:0/2704094004 wait complete. 2024-03-25T19:47:21.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:21.937+0000 7faeb1d93640 1 librados: shutdown 2024-03-25T19:47:21.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:28.754 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.868+0000 7faa90a6a640 10 monclient: get_monmap_and_config 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.868+0000 7faa90a6a640 10 monclient: build_initial_monmap 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.868+0000 7faa90a6a640 10 monclient: monmap: 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:47:28.869064+0000 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:47:28.869064+0000 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:28.868 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:47:28.869 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:47:28.869 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.868+0000 7faa90a6a640 1 Processor -- start 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 1 -- start start 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 10 monclient: init 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 10 monclient(hunting): picked mon.noname-c con 0x7faa8c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:28.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c142520 con 0x7faa8c13f710 2024-03-25T19:47:28.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0b28a0 con 0x7faa8c0a98f0 2024-03-25T19:47:28.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0b2a30 con 0x7faa8c1430c0 2024-03-25T19:47:28.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:28.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.869+0000 7faa90a6a640 10 monclient(hunting): authenticate will time out at 5275.996661s 2024-03-25T19:47:28.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.880+0000 7faa8a59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faa8c0a98f0 0x7faa8c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:43250/0 (socket says 172.21.15.18:43250) 2024-03-25T19:47:28.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.880+0000 7faa8a59c640 1 -- 172.21.15.18:0/1114687819 learned_addr learned my addr 172.21.15.18:0/1114687819 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:28.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.899+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267102772 0 0) 0x7faa8c142520 con 0x7faa8c13f710 2024-03-25T19:47:28.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.899+0000 7faa8959a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.899+0000 7faa8959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.899+0000 7faa8959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.899+0000 7faa8959a640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:47:28.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.899+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa74003520 con 0x7faa8c13f710 2024-03-25T19:47:28.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.962+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261409192 0 0) 0x7faa8c0b28a0 con 0x7faa8c0a98f0 2024-03-25T19:47:28.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.962+0000 7faa8959a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.962+0000 7faa8959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.962+0000 7faa8959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.962+0000 7faa8959a640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:47:28.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.962+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c142520 con 0x7faa8c0a98f0 2024-03-25T19:47:28.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.969+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667056096 0 0) 0x7faa8c0b2a30 con 0x7faa8c1430c0 2024-03-25T19:47:28.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.969+0000 7faa8959a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.969+0000 7faa8959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.969+0000 7faa8959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.969+0000 7faa8959a640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:47:28.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.969+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c0b28a0 con 0x7faa8c1430c0 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 26834439 0 0) 0x7faa74003520 con 0x7faa8c13f710 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 >> v1:172.21.15.18:6790/0 conn(0x7faa8c1430c0 legacy=0x7faa8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 >> v1:172.21.15.53:6789/0 conn(0x7faa8c0a98f0 legacy=0x7faa8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 10 monclient: _finish_hunting 0 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 1 monclient: found mon.noname-a 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0b44c0 con 0x7faa8c13f710 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 10 monclient: _finish_auth 0 2024-03-25T19:47:28.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 10 monclient: _check_auth_tickets 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faa80001690 con 0x7faa8c13f710 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa90a6a640 5 monclient: authenticate success, global_id 4524 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa90a6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa8959a640 10 monclient: dump: 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:28.976 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:28.977 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:28.977 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:28.977 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:28.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.975+0000 7faa90a6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:28.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.997+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa800017f0 con 0x7faa8c13f710 2024-03-25T19:47:28.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.997+0000 7faa8959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:28.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.997+0000 7faa8959a640 1 -- 172.21.15.18:0/1114687819 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faa80001c00 con 0x7faa8c13f710 2024-03-25T19:47:28.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.997+0000 7faa90a6a640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:28.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.997+0000 7faa90a6a640 4 set_mon_vals no callback set 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.997+0000 7faa8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.997+0000 7faa8959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.997+0000 7faa8959a640 10 monclient: dump: 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:28.998 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.998+0000 7faa90a6a640 10 monclient: shutdown 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.998+0000 7faa90a6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.998+0000 7faa90a6a640 1 -- 172.21.15.18:0/1114687819 >> v1:172.21.15.18:6789/0 conn(0x7faa8c13f710 legacy=0x7faa8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.998+0000 7faa90a6a640 1 -- 172.21.15.18:0/1114687819 shutdown_connections 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.998+0000 7faa90a6a640 1 -- 172.21.15.18:0/1114687819 >> 172.21.15.18:0/1114687819 conn(0x7faa8c0b7ea0 msgr2=0x7faa8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.998+0000 7faa90a6a640 1 -- 172.21.15.18:0/1114687819 shutdown_connections 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.998+0000 7faa90a6a640 1 -- 172.21.15.18:0/1114687819 wait complete. 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.999+0000 7faa90a6a640 10 monclient: build_initial_monmap 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:28.999+0000 7faa90a6a640 10 monclient: monmap: 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 1 librados: starting msgr at 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 1 librados: starting objecter 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 1 Processor -- start 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 1 -- start start 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 1 librados: setting wanted keys 2024-03-25T19:47:29.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 1 librados: calling monclient init 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 10 monclient: init 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 10 monclient(hunting): picked mon.noname-a con 0x7faa8c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.000+0000 7faa90a6a640 10 monclient(hunting): picked mon.noname-b con 0x7faa8c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.001+0000 7faa90a6a640 10 monclient(hunting): picked mon.noname-c con 0x7faa8c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.001+0000 7faa90a6a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0abac0 con 0x7faa8c0a98f0 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.001+0000 7faa90a6a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c0abda0 con 0x7faa8c13f710 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.001+0000 7faa90a6a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c1380e0 con 0x7faa8c1430c0 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.001+0000 7faa90a6a640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:29.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.001+0000 7faa90a6a640 10 monclient(hunting): authenticate will time out at 5276.127798s 2024-03-25T19:47:29.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.008+0000 7faa89d9b640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faa8c13f710 0x7faa8c138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:43304/0 (socket says 172.21.15.18:43304) 2024-03-25T19:47:29.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.008+0000 7faa89d9b640 1 -- 172.21.15.18:0/1716923257 learned_addr learned my addr 172.21.15.18:0/1716923257 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:29.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.009+0000 7faa7affd640 10 client.?.objecter ms_handle_connect 0x7faa8c1430c0 2024-03-25T19:47:29.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.009+0000 7faa7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:29.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.017+0000 7faa7affd640 10 client.?.objecter ms_handle_connect 0x7faa8c13f710 2024-03-25T19:47:29.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.017+0000 7faa7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:29.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.026+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351247713 0 0) 0x7faa8c1380e0 con 0x7faa8c1430c0 2024-03-25T19:47:29.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.026+0000 7faa7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.026+0000 7faa7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.026+0000 7faa7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.026+0000 7faa7affd640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:47:29.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.026+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6c004eb0 con 0x7faa8c1430c0 2024-03-25T19:47:29.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.030+0000 7faa7affd640 10 client.?.objecter ms_handle_connect 0x7faa8c0a98f0 2024-03-25T19:47:29.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.030+0000 7faa7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:29.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.041+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118885249 0 0) 0x7faa6c004eb0 con 0x7faa8c1430c0 2024-03-25T19:47:29.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.041+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa8c1380e0 con 0x7faa8c1430c0 2024-03-25T19:47:29.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.041+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faa84001bd0 con 0x7faa8c1430c0 2024-03-25T19:47:29.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.041+0000 7faa7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:29.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.041+0000 7faa7affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:29.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.041+0000 7faa7affd640 10 monclient(hunting): dump: 2024-03-25T19:47:29.041 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:29.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.041+0000 7faa7affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:29.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.046+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490887223 0 0) 0x7faa8c0abda0 con 0x7faa8c13f710 2024-03-25T19:47:29.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.047+0000 7faa7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.047+0000 7faa7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.047+0000 7faa7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.047+0000 7faa7affd640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:47:29.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.047+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6c004eb0 con 0x7faa8c13f710 2024-03-25T19:47:29.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.055+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494396910 0 0) 0x7faa8c0abac0 con 0x7faa8c0a98f0 2024-03-25T19:47:29.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.056+0000 7faa7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:29.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.056+0000 7faa7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:29.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.056+0000 7faa7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:29.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.056+0000 7faa7affd640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:47:29.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.056+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8c0abda0 con 0x7faa8c0a98f0 2024-03-25T19:47:29.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3527360211 0 0) 0x7faa8c1380e0 con 0x7faa8c1430c0 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 >> v1:172.21.15.53:6789/0 conn(0x7faa8c13f710 legacy=0x7faa8c138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 >> v1:172.21.15.18:6789/0 conn(0x7faa8c0a98f0 legacy=0x7faa8c0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 1 monclient: found mon.c 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c137d90 con 0x7faa8c1430c0 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 10 monclient: _finish_auth 0 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 5 monclient: authenticate success, global_id 4493 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 10 monclient: _renew_subs 2024-03-25T19:47:29.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:29.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c139240 con 0x7faa8c1430c0 2024-03-25T19:47:29.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:29.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 10 monclient: _renew_subs 2024-03-25T19:47:29.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:29.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c136830 con 0x7faa8c1430c0 2024-03-25T19:47:29.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.057+0000 7faa90a6a640 1 librados: init done 2024-03-25T19:47:29.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.058+0000 7faa90a6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:29.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.058+0000 7faa90a6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:29.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.058+0000 7faa90a6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:29.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.058+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c0627b0 con 0x7faa8c1430c0 2024-03-25T19:47:29.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.271+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa840020a0 con 0x7faa8c1430c0 2024-03-25T19:47:29.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.271+0000 7faa7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:29.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.271+0000 7faa88d99640 4 set_mon_vals no callback set 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.363+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faa84003b30 con 0x7faa8c1430c0 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.363+0000 7faa7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.363+0000 7faa7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.363+0000 7faa7affd640 10 monclient: dump: 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:29.363 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:29.364 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:29.364 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:29.364 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:29.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.364+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7faa84003d50 con 0x7faa8c1430c0 2024-03-25T19:47:29.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.364+0000 7faa8a59c640 10 monclient: get_auth_request con 0x7faa6c06de10 auth_method 0 2024-03-25T19:47:29.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.365+0000 7faa7affd640 10 client.4493.objecter ms_handle_connect 0x7faa6c06de10 2024-03-25T19:47:29.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.372+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7faa840539a0 con 0x7faa8c1430c0 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.372+0000 7faa7affd640 10 client.4493.objecter ms_dispatch 0x7faa8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.372+0000 7faa7affd640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.372+0000 7faa7affd640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.373+0000 7faa7affd640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.373+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa8404f2e0 con 0x7faa8c1430c0 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.373+0000 7faa7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:29.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.373+0000 7faa7affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:29.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.626+0000 7faa90a6a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.626+0000 7faa90a6a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.626+0000 7faa90a6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:29.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.626+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa8c13f070 con 0x7faa8c1430c0 2024-03-25T19:47:29.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.655+0000 7faa7affd640 1 -- 172.21.15.18:0/1716923257 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7faa840513e0 con 0x7faa8c1430c0 2024-03-25T19:47:29.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.655+0000 7faa7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.655+0000 7faa7affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:29.656 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:47:29.656 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:29.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.657+0000 7faa90a6a640 10 librados: watch_flush enter 2024-03-25T19:47:29.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 10 librados: watch_flush exit 2024-03-25T19:47:29.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-25T19:47:29.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 10 client.4493.objecter successfully canceled tick 2024-03-25T19:47:29.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 >> v1:172.21.15.18:6800/1520000573 conn(0x7faa6c06de10 legacy=0x7faa6c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 10 monclient: shutdown 2024-03-25T19:47:29.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:29.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 >> v1:172.21.15.18:6790/0 conn(0x7faa8c1430c0 legacy=0x7faa8c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 shutdown_connections 2024-03-25T19:47:29.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 >> 172.21.15.18:0/1716923257 conn(0x7faa8c0b7ea0 msgr2=0x7faa8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:29.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 shutdown_connections 2024-03-25T19:47:29.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 1 -- 172.21.15.18:0/1716923257 wait complete. 2024-03-25T19:47:29.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:29.658+0000 7faa90a6a640 1 librados: shutdown 2024-03-25T19:47:29.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:33.154 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:00.684 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.161 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:00.691 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.197 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:00.727 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.217 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:00.713 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.220 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:00.751 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.233 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:00.764 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.234 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:00.731 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.263 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:00.759 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:33.263 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:00.794 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:36.477 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:36.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.592+0000 7f82d651c640 10 monclient: get_monmap_and_config 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.592+0000 7f82d651c640 10 monclient: build_initial_monmap 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.592+0000 7f82d651c640 10 monclient: monmap: 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:47:36.593603+0000 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:47:36.593603+0000 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:47:36.593 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.593+0000 7f82d651c640 1 Processor -- start 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.593+0000 7f82d651c640 1 -- start start 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.593+0000 7f82d651c640 10 monclient: init 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 10 monclient(hunting): picked mon.noname-a con 0x7f82d00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 10 monclient(hunting): picked mon.noname-b con 0x7f82d013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 10 monclient(hunting): picked mon.noname-c con 0x7f82d01430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0142520 con 0x7f82d00a98f0 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00b28a0 con 0x7f82d01430c0 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00b2a30 con 0x7f82d013f710 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:36.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.594+0000 7f82d651c640 10 monclient(hunting): authenticate will time out at 5283.721580s 2024-03-25T19:47:36.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.608+0000 7f82d4ab9640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f82d01430c0 0x7f82d014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:44602/0 (socket says 172.21.15.18:44602) 2024-03-25T19:47:36.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.608+0000 7f82d4ab9640 1 -- 172.21.15.18:0/2258578719 learned_addr learned my addr 172.21.15.18:0/2258578719 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.624+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207007641 0 0) 0x7f82d00b28a0 con 0x7f82d01430c0 2024-03-25T19:47:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.624+0000 7f82ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.624+0000 7f82ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.624+0000 7f82ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.624+0000 7f82ceffd640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:47:36.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.624+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b8003520 con 0x7f82d01430c0 2024-03-25T19:47:36.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.625+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223517504 0 0) 0x7f82d0142520 con 0x7f82d00a98f0 2024-03-25T19:47:36.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.625+0000 7f82ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.625+0000 7f82ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.625+0000 7f82ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.625+0000 7f82ceffd640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:47:36.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.625+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d00b28a0 con 0x7f82d00a98f0 2024-03-25T19:47:36.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.637+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616412201 0 0) 0x7f82d00b2a30 con 0x7f82d013f710 2024-03-25T19:47:36.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.637+0000 7f82ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.637+0000 7f82ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.637+0000 7f82ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.637+0000 7f82ceffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:47:36.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.637+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d0142520 con 0x7f82d013f710 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3719293506 0 0) 0x7f82b8003520 con 0x7f82d01430c0 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 >> v1:172.21.15.18:6790/0 conn(0x7f82d013f710 legacy=0x7f82d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 >> v1:172.21.15.18:6789/0 conn(0x7f82d00a98f0 legacy=0x7f82d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 1 monclient: found mon.noname-c 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d00b44c0 con 0x7f82d01430c0 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f82c8001a10 con 0x7f82d01430c0 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.641+0000 7f82ceffd640 10 monclient: dump: 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.642+0000 7f82d651c640 5 monclient: authenticate success, global_id 4495 2024-03-25T19:47:36.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.642+0000 7f82d651c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.665+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c8001b70 con 0x7f82d01430c0 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.665+0000 7f82ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.665+0000 7f82ceffd640 1 -- 172.21.15.18:0/2258578719 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f82c8001f80 con 0x7f82d01430c0 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.665+0000 7f82ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.665+0000 7f82ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.665+0000 7f82ceffd640 10 monclient: dump: 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:36.665 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.666 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:36.666 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:36.666 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:36.666 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:36.666 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:36.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.665+0000 7f82d651c640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:36.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.665+0000 7f82d651c640 4 set_mon_vals no callback set 2024-03-25T19:47:36.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 10 monclient: shutdown 2024-03-25T19:47:36.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:36.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 1 -- 172.21.15.18:0/2258578719 >> v1:172.21.15.53:6789/0 conn(0x7f82d01430c0 legacy=0x7f82d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 1 -- 172.21.15.18:0/2258578719 shutdown_connections 2024-03-25T19:47:36.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 1 -- 172.21.15.18:0/2258578719 >> 172.21.15.18:0/2258578719 conn(0x7f82d00b7ea0 msgr2=0x7f82d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:36.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 1 -- 172.21.15.18:0/2258578719 shutdown_connections 2024-03-25T19:47:36.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 1 -- 172.21.15.18:0/2258578719 wait complete. 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 10 monclient: build_initial_monmap 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.666+0000 7f82d651c640 10 monclient: monmap: 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.667+0000 7f82d651c640 1 librados: starting msgr at 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.667+0000 7f82d651c640 1 librados: starting objecter 2024-03-25T19:47:36.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.667+0000 7f82d651c640 1 Processor -- start 2024-03-25T19:47:36.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.667+0000 7f82d651c640 1 -- start start 2024-03-25T19:47:36.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.667+0000 7f82d651c640 1 librados: setting wanted keys 2024-03-25T19:47:36.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.667+0000 7f82d651c640 1 librados: calling monclient init 2024-03-25T19:47:36.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.667+0000 7f82d651c640 10 monclient: init 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 10 monclient(hunting): picked mon.noname-b con 0x7f82d00a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 10 monclient(hunting): picked mon.noname-c con 0x7f82d013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 10 monclient(hunting): picked mon.noname-a con 0x7f82d01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d01369d0 con 0x7f82d01430c0 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0133a60 con 0x7f82d00a98f0 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00ab9e0 con 0x7f82d013f710 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:36.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.668+0000 7f82d651c640 10 monclient(hunting): authenticate will time out at 5283.795228s 2024-03-25T19:47:36.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.686+0000 7f82cffff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f82d00a98f0 0x7f82d0132c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:44656/0 (socket says 172.21.15.18:44656) 2024-03-25T19:47:36.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.686+0000 7f82cffff640 1 -- 172.21.15.18:0/2923115813 learned_addr learned my addr 172.21.15.18:0/2923115813 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:36.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.687+0000 7f82ccff9640 10 client.?.objecter ms_handle_connect 0x7f82d00a98f0 2024-03-25T19:47:36.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.687+0000 7f82ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:36.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.694+0000 7f82ccff9640 10 client.?.objecter ms_handle_connect 0x7f82d01430c0 2024-03-25T19:47:36.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.694+0000 7f82ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:36.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.699+0000 7f82ccff9640 10 client.?.objecter ms_handle_connect 0x7f82d013f710 2024-03-25T19:47:36.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.699+0000 7f82ccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:36.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.704+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368281116 0 0) 0x7f82d0133a60 con 0x7f82d00a98f0 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.704+0000 7f82ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.704+0000 7f82ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.704+0000 7f82ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.704+0000 7f82ccff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.704+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc004eb0 con 0x7f82d00a98f0 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.705+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100755942 0 0) 0x7f82d00ab9e0 con 0x7f82d013f710 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.705+0000 7f82ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.705+0000 7f82ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.705+0000 7f82ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.705+0000 7f82ccff9640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:47:36.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.705+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d0133a60 con 0x7f82d013f710 2024-03-25T19:47:36.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.715+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890220522 0 0) 0x7f82d01369d0 con 0x7f82d01430c0 2024-03-25T19:47:36.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.715+0000 7f82ccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.715+0000 7f82ccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.715+0000 7f82ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.715+0000 7f82ccff9640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:47:36.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.715+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d00ab9e0 con 0x7f82d01430c0 2024-03-25T19:47:36.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.723+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1744513069 0 0) 0x7f82bc004eb0 con 0x7f82d00a98f0 2024-03-25T19:47:36.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.723+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82d01369d0 con 0x7f82d00a98f0 2024-03-25T19:47:36.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.723+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f82c0001a30 con 0x7f82d00a98f0 2024-03-25T19:47:36.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.723+0000 7f82ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.723+0000 7f82ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:36.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.723+0000 7f82ccff9640 10 monclient(hunting): dump: 2024-03-25T19:47:36.723 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:36.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.723+0000 7f82ccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.725+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619825783 0 0) 0x7f82d0133a60 con 0x7f82d013f710 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.725+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc004eb0 con 0x7f82d013f710 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.725+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f82c4001870 con 0x7f82d013f710 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.725+0000 7f82ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.725+0000 7f82ccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.725+0000 7f82ccff9640 10 monclient(hunting): dump: 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:36.725 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:36.726 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:36.726 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:36.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1011574820 0 0) 0x7f82d01369d0 con 0x7f82d00a98f0 2024-03-25T19:47:36.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 >> v1:172.21.15.18:6790/0 conn(0x7f82d013f710 legacy=0x7f82d01362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 >> v1:172.21.15.18:6789/0 conn(0x7f82d01430c0 legacy=0x7f82d0133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 1 monclient: found mon.b 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d00abcc0 con 0x7f82d00a98f0 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 10 monclient: _renew_subs 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d00abfa0 con 0x7f82d00a98f0 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 10 monclient: _renew_subs 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:36.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d0137e40 con 0x7f82d00a98f0 2024-03-25T19:47:36.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.739+0000 7f82d651c640 1 librados: init done 2024-03-25T19:47:36.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.740+0000 7f82d651c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:36.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.740+0000 7f82d651c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:36.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.740+0000 7f82d651c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:36.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:36.740+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d00627b0 con 0x7f82d00a98f0 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.032+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c0001c70 con 0x7f82d00a98f0 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.032+0000 7f82ccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.032+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f82c0003a60 con 0x7f82d00a98f0 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.032+0000 7f82ccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.032+0000 7f82ccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.032+0000 7f82ccff9640 10 monclient: dump: 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:37.032 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.032+0000 7f82ce7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:37.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.033+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f82c0003c80 con 0x7f82d00a98f0 2024-03-25T19:47:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.033+0000 7f82cf7fe640 10 monclient: get_auth_request con 0x7f82bc06df70 auth_method 0 2024-03-25T19:47:37.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.034+0000 7f82ccff9640 10 client.4498.objecter ms_handle_connect 0x7f82bc06df70 2024-03-25T19:47:37.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.042+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f82c0054870 con 0x7f82d00a98f0 2024-03-25T19:47:37.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.042+0000 7f82ccff9640 10 client.4498.objecter ms_dispatch 0x7f82d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:37.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.042+0000 7f82ccff9640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:37.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.042+0000 7f82ccff9640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:37.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.042+0000 7f82ccff9640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T19:47:37.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.042+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82c0050380 con 0x7f82d00a98f0 2024-03-25T19:47:37.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.042+0000 7f82ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:37.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.042+0000 7f82ccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:37.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.279+0000 7f82d651c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:37.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.279+0000 7f82d651c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:37.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.279+0000 7f82d651c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:37.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.279+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82d013dfd0 con 0x7f82d00a98f0 2024-03-25T19:47:37.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.305+0000 7f82ccff9640 1 -- 172.21.15.18:0/2923115813 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f82c00522b0 con 0x7f82d00a98f0 2024-03-25T19:47:37.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.305+0000 7f82ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:37.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.305+0000 7f82ccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:37.305 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:47:37.305 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:37.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 10 librados: watch_flush enter 2024-03-25T19:47:37.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 10 librados: watch_flush exit 2024-03-25T19:47:37.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T19:47:37.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 10 client.4498.objecter successfully canceled tick 2024-03-25T19:47:37.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 >> v1:172.21.15.18:6800/1520000573 conn(0x7f82bc06df70 legacy=0x7f82bc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:37.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 10 monclient: shutdown 2024-03-25T19:47:37.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:37.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 >> v1:172.21.15.53:6789/0 conn(0x7f82d00a98f0 legacy=0x7f82d0132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:37.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 shutdown_connections 2024-03-25T19:47:37.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 >> 172.21.15.18:0/2923115813 conn(0x7f82d00b7ea0 msgr2=0x7f82d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:37.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 shutdown_connections 2024-03-25T19:47:37.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 1 -- 172.21.15.18:0/2923115813 wait complete. 2024-03-25T19:47:37.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:37.307+0000 7f82d651c640 1 librados: shutdown 2024-03-25T19:47:37.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:44.126 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:44.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.235+0000 7fe250e40640 10 monclient: get_monmap_and_config 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.235+0000 7fe250e40640 10 monclient: build_initial_monmap 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.235+0000 7fe250e40640 10 monclient: monmap: 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:47:44.236752+0000 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:47:44.236752+0000 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:47:44.236 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:44.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.236+0000 7fe250e40640 1 Processor -- start 2024-03-25T19:47:44.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.236+0000 7fe250e40640 1 -- start start 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.236+0000 7fe250e40640 10 monclient: init 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 10 monclient(hunting): picked mon.noname-a con 0x7fe24c061830 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 10 monclient(hunting): picked mon.noname-c con 0x7fe24c13d9f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 10 monclient(hunting): picked mon.noname-b con 0x7fe24c143d50 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c071af0 con 0x7fe24c061830 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c1441c0 con 0x7fe24c13d9f0 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c072810 con 0x7fe24c143d50 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:44.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.237+0000 7fe250e40640 10 monclient(hunting): authenticate will time out at 5291.364490s 2024-03-25T19:47:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.249+0000 7fe24a59c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe24c061830 0x7fe24c13d2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:56168/0 (socket says 172.21.15.18:56168) 2024-03-25T19:47:44.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.249+0000 7fe24a59c640 1 -- 172.21.15.18:0/1190429488 learned_addr learned my addr 172.21.15.18:0/1190429488 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:44.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.261+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312670823 0 0) 0x7fe24c071af0 con 0x7fe24c061830 2024-03-25T19:47:44.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.261+0000 7fe24959a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.261+0000 7fe24959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.261+0000 7fe24959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.261+0000 7fe24959a640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:47:44.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.261+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe234003520 con 0x7fe24c061830 2024-03-25T19:47:44.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.269+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930786173 0 0) 0x7fe24c072810 con 0x7fe24c143d50 2024-03-25T19:47:44.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.269+0000 7fe24959a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.269+0000 7fe24959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.269+0000 7fe24959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.269+0000 7fe24959a640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:47:44.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.269+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe24c071af0 con 0x7fe24c143d50 2024-03-25T19:47:44.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4064968774 0 0) 0x7fe234003520 con 0x7fe24c061830 2024-03-25T19:47:44.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 >> v1:172.21.15.18:6790/0 conn(0x7fe24c143d50 legacy=0x7fe24c14b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 >> v1:172.21.15.53:6789/0 conn(0x7fe24c13d9f0 legacy=0x7fe24c143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 10 monclient: _finish_hunting 0 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 1 monclient: found mon.noname-a 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c072af0 con 0x7fe24c061830 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 10 monclient: _finish_auth 0 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 10 monclient: _check_auth_tickets 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe23c001690 con 0x7fe24c061830 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe24959a640 10 monclient: dump: 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:44.276 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe250e40640 5 monclient: authenticate success, global_id 4536 2024-03-25T19:47:44.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.275+0000 7fe250e40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:44.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.281+0000 7fe24959a640 1 -- 172.21.15.18:0/1190429488 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe23c0017f0 con 0x7fe24c061830 2024-03-25T19:47:44.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.281+0000 7fe24959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:44.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.281+0000 7fe250e40640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:44.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.281+0000 7fe250e40640 4 set_mon_vals no callback set 2024-03-25T19:47:44.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.282+0000 7fe250e40640 10 monclient: shutdown 2024-03-25T19:47:44.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.282+0000 7fe250e40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:44.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.282+0000 7fe250e40640 1 -- 172.21.15.18:0/1190429488 >> v1:172.21.15.18:6789/0 conn(0x7fe24c061830 legacy=0x7fe24c13d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.282+0000 7fe250e40640 1 -- 172.21.15.18:0/1190429488 shutdown_connections 2024-03-25T19:47:44.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.282+0000 7fe250e40640 1 -- 172.21.15.18:0/1190429488 >> 172.21.15.18:0/1190429488 conn(0x7fe24c0b11e0 msgr2=0x7fe24c0b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:44.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.282+0000 7fe250e40640 1 -- 172.21.15.18:0/1190429488 shutdown_connections 2024-03-25T19:47:44.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.282+0000 7fe250e40640 1 -- 172.21.15.18:0/1190429488 wait complete. 2024-03-25T19:47:44.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.283+0000 7fe250e40640 10 monclient: build_initial_monmap 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.283+0000 7fe250e40640 10 monclient: monmap: 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.283+0000 7fe250e40640 1 librados: starting msgr at 2024-03-25T19:47:44.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.283+0000 7fe250e40640 1 librados: starting objecter 2024-03-25T19:47:44.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 1 Processor -- start 2024-03-25T19:47:44.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 1 -- start start 2024-03-25T19:47:44.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 1 librados: setting wanted keys 2024-03-25T19:47:44.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 1 librados: calling monclient init 2024-03-25T19:47:44.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 10 monclient: init 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 10 monclient(hunting): picked mon.noname-b con 0x7fe24c061830 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 10 monclient(hunting): picked mon.noname-c con 0x7fe24c13d9f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 10 monclient(hunting): picked mon.noname-a con 0x7fe24c143d50 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c13a3a0 con 0x7fe24c143d50 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c13a120 con 0x7fe24c061830 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c1c9830 con 0x7fe24c13d9f0 2024-03-25T19:47:44.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:44.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.284+0000 7fe250e40640 10 monclient(hunting): authenticate will time out at 5291.411671s 2024-03-25T19:47:44.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.288+0000 7fe249d9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fe24c13d9f0 0x7fe24c139300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49460/0 (socket says 172.21.15.18:49460) 2024-03-25T19:47:44.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.288+0000 7fe249d9b640 1 -- 172.21.15.18:0/2852010062 learned_addr learned my addr 172.21.15.18:0/2852010062 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:44.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.293+0000 7fe23affd640 10 client.?.objecter ms_handle_connect 0x7fe24c061830 2024-03-25T19:47:44.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.293+0000 7fe23affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.299+0000 7fe23affd640 10 client.?.objecter ms_handle_connect 0x7fe24c143d50 2024-03-25T19:47:44.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.299+0000 7fe23affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.303+0000 7fe23affd640 10 client.?.objecter ms_handle_connect 0x7fe24c13d9f0 2024-03-25T19:47:44.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.303+0000 7fe23affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.308+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203491159 0 0) 0x7fe24c13a120 con 0x7fe24c061830 2024-03-25T19:47:44.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.308+0000 7fe23affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.308+0000 7fe23affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.308+0000 7fe23affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.308+0000 7fe23affd640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:47:44.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.308+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe22c004eb0 con 0x7fe24c061830 2024-03-25T19:47:44.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.309+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408055753 0 0) 0x7fe24c1c9830 con 0x7fe24c13d9f0 2024-03-25T19:47:44.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.309+0000 7fe23affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.309+0000 7fe23affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.309+0000 7fe23affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.309+0000 7fe23affd640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:47:44.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.309+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe24c13a120 con 0x7fe24c13d9f0 2024-03-25T19:47:44.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.310+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784895562 0 0) 0x7fe24c071af0 con 0x7fe24c143d50 2024-03-25T19:47:44.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.310+0000 7fe23affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.310+0000 7fe23affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.310+0000 7fe23affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.310+0000 7fe23affd640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:47:44.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.310+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe24c1c9830 con 0x7fe24c143d50 2024-03-25T19:47:44.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.320+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1791645392 0 0) 0x7fe22c004eb0 con 0x7fe24c061830 2024-03-25T19:47:44.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.320+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe24c071af0 con 0x7fe24c061830 2024-03-25T19:47:44.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.320+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe23c001ca0 con 0x7fe24c061830 2024-03-25T19:47:44.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.320+0000 7fe23affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.320+0000 7fe23affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.320+0000 7fe23affd640 10 monclient(hunting): dump: 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:44.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.320+0000 7fe23affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:44.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.325+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2034041082 0 0) 0x7fe24c13a120 con 0x7fe24c13d9f0 2024-03-25T19:47:44.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.325+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe22c004eb0 con 0x7fe24c13d9f0 2024-03-25T19:47:44.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.325+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe240001970 con 0x7fe24c13d9f0 2024-03-25T19:47:44.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.325+0000 7fe23affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.325+0000 7fe23affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.325+0000 7fe23affd640 10 monclient(hunting): dump: 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:44.326 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.327+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3410132608 0 0) 0x7fe24c1c9830 con 0x7fe24c143d50 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.327+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe24c13a120 con 0x7fe24c143d50 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.327+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe244001a30 con 0x7fe24c143d50 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.327+0000 7fe23affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.327+0000 7fe23affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.327+0000 7fe23affd640 10 monclient(hunting): dump: 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.327 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.328 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.328 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:44.328 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:44.328 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:44.328 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:44.328 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:44.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2952581090 0 0) 0x7fe24c071af0 con 0x7fe24c061830 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 >> v1:172.21.15.18:6790/0 conn(0x7fe24c13d9f0 legacy=0x7fe24c139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 >> v1:172.21.15.18:6789/0 conn(0x7fe24c143d50 legacy=0x7fe24c139a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 1 monclient: found mon.b 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c13a560 con 0x7fe24c061830 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 10 monclient: _finish_auth 0 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe23affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 5 monclient: authenticate success, global_id 4504 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 10 monclient: _renew_subs 2024-03-25T19:47:44.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:44.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c1cc0e0 con 0x7fe24c061830 2024-03-25T19:47:44.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:44.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 10 monclient: _renew_subs 2024-03-25T19:47:44.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:44.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c0a9c50 con 0x7fe24c061830 2024-03-25T19:47:44.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.344+0000 7fe250e40640 1 librados: init done 2024-03-25T19:47:44.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.345+0000 7fe250e40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:44.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.345+0000 7fe250e40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:44.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.345+0000 7fe250e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:44.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.345+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24c0627b0 con 0x7fe24c061830 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.620+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe23c004a90 con 0x7fe24c061830 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.620+0000 7fe23affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.620+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fe23c004da0 con 0x7fe24c061830 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.620+0000 7fe23affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.620+0000 7fe23affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.620+0000 7fe23affd640 10 monclient: dump: 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.620 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:44.621 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.621 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:44.621 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:44.621 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:44.621 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:44.621 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:44.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.620+0000 7fe248d99640 4 set_mon_vals no callback set 2024-03-25T19:47:44.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.621+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fe23c01ad70 con 0x7fe24c061830 2024-03-25T19:47:44.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.622+0000 7fe249d9b640 10 monclient: get_auth_request con 0x7fe22c06e0f0 auth_method 0 2024-03-25T19:47:44.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.622+0000 7fe23affd640 10 client.4504.objecter ms_handle_connect 0x7fe22c06e0f0 2024-03-25T19:47:44.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.626+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fe23c054870 con 0x7fe24c061830 2024-03-25T19:47:44.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.626+0000 7fe23affd640 10 client.4504.objecter ms_dispatch 0x7fe24c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:44.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.626+0000 7fe23affd640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:44.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.626+0000 7fe23affd640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:44.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.626+0000 7fe23affd640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T19:47:44.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.626+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe23c050380 con 0x7fe24c061830 2024-03-25T19:47:44.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.626+0000 7fe23affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:44.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.626+0000 7fe23affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:44.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.886+0000 7fe250e40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:44.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.886+0000 7fe250e40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:44.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.886+0000 7fe250e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:44.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.886+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe24c1421d0 con 0x7fe24c061830 2024-03-25T19:47:44.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.916+0000 7fe23affd640 1 -- 172.21.15.18:0/2852010062 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fe23c0522b0 con 0x7fe24c061830 2024-03-25T19:47:44.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.916+0000 7fe23affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:44.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.916+0000 7fe23affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:44.916 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:47:44.917 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:44.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.918+0000 7fe250e40640 10 librados: watch_flush enter 2024-03-25T19:47:44.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.918+0000 7fe250e40640 10 librados: watch_flush exit 2024-03-25T19:47:44.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T19:47:44.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 10 client.4504.objecter successfully canceled tick 2024-03-25T19:47:44.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 >> v1:172.21.15.18:6800/1520000573 conn(0x7fe22c06e0f0 legacy=0x7fe22c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 10 monclient: shutdown 2024-03-25T19:47:44.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:44.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 >> v1:172.21.15.53:6789/0 conn(0x7fe24c061830 legacy=0x7fe24c1d8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 shutdown_connections 2024-03-25T19:47:44.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 >> 172.21.15.18:0/2852010062 conn(0x7fe24c0b11e0 msgr2=0x7fe24c0afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:44.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 shutdown_connections 2024-03-25T19:47:44.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 1 -- 172.21.15.18:0/2852010062 wait complete. 2024-03-25T19:47:44.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:44.919+0000 7fe250e40640 1 librados: shutdown 2024-03-25T19:47:44.930 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:51.737 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.857+0000 7f8cc444e640 10 monclient: get_monmap_and_config 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.857+0000 7f8cc444e640 10 monclient: build_initial_monmap 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.857+0000 7f8cc444e640 10 monclient: monmap: 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:47:51.857985+0000 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:47:51.857985+0000 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:47:51.857 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:47:51.858 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.857+0000 7f8cc444e640 1 Processor -- start 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.857+0000 7f8cc444e640 1 -- start start 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.857+0000 7f8cc444e640 10 monclient: init 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 10 monclient(hunting): picked mon.noname-a con 0x7f8cbc06c380 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 10 monclient(hunting): picked mon.noname-c con 0x7f8cbc0b12d0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 10 monclient(hunting): picked mon.noname-b con 0x7f8cbc0aedf0 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0b16f0 con 0x7f8cbc06c380 2024-03-25T19:47:51.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0adfe0 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0add00 con 0x7f8cbc0aedf0 2024-03-25T19:47:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.858+0000 7f8cc444e640 10 monclient(hunting): authenticate will time out at 5298.985710s 2024-03-25T19:47:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.861+0000 7f8cc19e9640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8cbc0b12d0 0x7f8cbc0ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:51030/0 (socket says 172.21.15.18:51030) 2024-03-25T19:47:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.861+0000 7f8cc19e9640 1 -- 172.21.15.18:0/757790351 learned_addr learned my addr 172.21.15.18:0/757790351 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:51.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.869+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75604704 0 0) 0x7f8cbc0adfe0 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.869+0000 7f8cc11e8640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.869+0000 7f8cc11e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.869+0000 7f8cc11e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.869+0000 7f8cc11e8640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:47:51.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.869+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca8003520 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899216709 0 0) 0x7f8ca8003520 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 >> v1:172.21.15.18:6790/0 conn(0x7f8cbc0aedf0 legacy=0x7f8cbc14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 >> v1:172.21.15.18:6789/0 conn(0x7f8cbc06c380 legacy=0x7f8cbc0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 10 monclient: _finish_hunting 0 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 1 monclient: found mon.noname-c 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc0ae2c0 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 10 monclient: _finish_auth 0 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 10 monclient: _check_auth_tickets 2024-03-25T19:47:51.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8cac001690 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc11e8640 10 monclient: dump: 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc444e640 5 monclient: authenticate success, global_id 4507 2024-03-25T19:47:51.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.879+0000 7f8cc444e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:51.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cac0017f0 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc11e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc11e8640 1 -- 172.21.15.18:0/757790351 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8cac001c40 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc11e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc11e8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc11e8640 10 monclient: dump: 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:51.892 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:51.893 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:51.893 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:51.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc444e640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:51.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc444e640 4 set_mon_vals no callback set 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc444e640 10 monclient: shutdown 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc444e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.892+0000 7f8cc444e640 1 -- 172.21.15.18:0/757790351 >> v1:172.21.15.53:6789/0 conn(0x7f8cbc0b12d0 legacy=0x7f8cbc0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.893+0000 7f8cc444e640 1 -- 172.21.15.18:0/757790351 shutdown_connections 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.893+0000 7f8cc444e640 1 -- 172.21.15.18:0/757790351 >> 172.21.15.18:0/757790351 conn(0x7f8cbc0b5d10 msgr2=0x7f8cbc0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.893+0000 7f8cc444e640 1 -- 172.21.15.18:0/757790351 shutdown_connections 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.893+0000 7f8cc444e640 1 -- 172.21.15.18:0/757790351 wait complete. 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 10 monclient: build_initial_monmap 2024-03-25T19:47:51.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 10 monclient: monmap: 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 1 librados: starting msgr at 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 1 librados: starting objecter 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 1 Processor -- start 2024-03-25T19:47:51.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 1 -- start start 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 1 librados: setting wanted keys 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 1 librados: calling monclient init 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.894+0000 7f8cc444e640 10 monclient: init 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 10 monclient(hunting): picked mon.noname-c con 0x7f8cbc06c380 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 10 monclient(hunting): picked mon.noname-b con 0x7f8cbc0aedf0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 10 monclient(hunting): picked mon.noname-a con 0x7f8cbc0b12d0 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc06c8e0 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc1a4e20 con 0x7f8cbc0aedf0 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc1a7350 con 0x7f8cbc06c380 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:51.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.895+0000 7f8cc444e640 10 monclient(hunting): authenticate will time out at 5299.022189s 2024-03-25T19:47:51.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.903+0000 7f8cc21ea640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f8cbc06c380 0x7f8cbc1c5e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51810/0 (socket says 172.21.15.18:51810) 2024-03-25T19:47:51.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.903+0000 7f8cc21ea640 1 -- 172.21.15.18:0/1744683740 learned_addr learned my addr 172.21.15.18:0/1744683740 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:51.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.904+0000 7f8cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f8cbc06c380 2024-03-25T19:47:51.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.904+0000 7f8cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:51.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.913+0000 7f8cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f8cbc0b12d0 2024-03-25T19:47:51.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.913+0000 7f8cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:51.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.916+0000 7f8cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f8cbc0aedf0 2024-03-25T19:47:51.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.916+0000 7f8cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:51.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.917+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324916206 0 0) 0x7f8cbc1a7350 con 0x7f8cbc06c380 2024-03-25T19:47:51.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.917+0000 7f8cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.917+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.917+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.917+0000 7f8cb2ffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:47:51.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.917+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca0004eb0 con 0x7f8cbc06c380 2024-03-25T19:47:51.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.925+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909016184 0 0) 0x7f8cbc06c8e0 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.925+0000 7f8cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.925+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.925+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.925+0000 7f8cb2ffd640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:47:51.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.925+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cbc1a7350 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.928+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1088208885 0 0) 0x7f8ca0004eb0 con 0x7f8cbc06c380 2024-03-25T19:47:51.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.928+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cbc06c8e0 con 0x7f8cbc06c380 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.928+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8cb8001960 con 0x7f8cbc06c380 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.928+0000 7f8cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.928+0000 7f8cb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.928+0000 7f8cb2ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:51.929 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:51.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.928+0000 7f8cb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:51.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655352491 0 0) 0x7f8cbc1a4e20 con 0x7f8cbc0aedf0 2024-03-25T19:47:51.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:47:51.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca0004eb0 con 0x7f8cbc0aedf0 2024-03-25T19:47:51.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1059322500 0 0) 0x7f8cbc1a7350 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cbc1a4e20 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8cb4001b10 con 0x7f8cbc0b12d0 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.943+0000 7f8cb2ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:51.944 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:51.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3602428003 0 0) 0x7f8cbc06c8e0 con 0x7f8cbc06c380 2024-03-25T19:47:51.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 >> v1:172.21.15.53:6789/0 conn(0x7f8cbc0aedf0 legacy=0x7f8cbc1d7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 >> v1:172.21.15.18:6789/0 conn(0x7f8cbc0b12d0 legacy=0x7f8cbc1d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 1 monclient: found mon.c 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc1c6520 con 0x7f8cbc06c380 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 5 monclient: authenticate success, global_id 4508 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 10 monclient: _renew_subs 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc1c67d0 con 0x7f8cbc06c380 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:51.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 10 monclient: _renew_subs 2024-03-25T19:47:51.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:51.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc1a5490 con 0x7f8cbc06c380 2024-03-25T19:47:51.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.952+0000 7f8cc444e640 1 librados: init done 2024-03-25T19:47:51.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.953+0000 7f8cc444e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:51.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.953+0000 7f8cc444e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:51.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.953+0000 7f8cc444e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:51.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:51.953+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cbc06d790 con 0x7f8cbc06c380 2024-03-25T19:47:52.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.261+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb80035f0 con 0x7f8cbc06c380 2024-03-25T19:47:52.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.261+0000 7f8cb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:52.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.261+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f8cb80037a0 con 0x7f8cbc06c380 2024-03-25T19:47:52.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.261+0000 7f8cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:52.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.261+0000 7f8cb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.261+0000 7f8cb2ffd640 10 monclient: dump: 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:52.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.261+0000 7f8cc09e7640 4 set_mon_vals no callback set 2024-03-25T19:47:52.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.262+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f8cb80039c0 con 0x7f8cbc06c380 2024-03-25T19:47:52.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.262+0000 7f8cc19e9640 10 monclient: get_auth_request con 0x7f8ca006dfb0 auth_method 0 2024-03-25T19:47:52.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.262+0000 7f8cb2ffd640 10 client.4508.objecter ms_handle_connect 0x7f8ca006dfb0 2024-03-25T19:47:52.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.270+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f8cb80545d0 con 0x7f8cbc06c380 2024-03-25T19:47:52.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.270+0000 7f8cb2ffd640 10 client.4508.objecter ms_dispatch 0x7f8cbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:52.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.270+0000 7f8cb2ffd640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:52.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.270+0000 7f8cb2ffd640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:52.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.271+0000 7f8cb2ffd640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-25T19:47:52.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.271+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cb80500e0 con 0x7f8cbc06c380 2024-03-25T19:47:52.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.271+0000 7f8cb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:52.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.271+0000 7f8cb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:52.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.511+0000 7f8cc444e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.511+0000 7f8cc444e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.511+0000 7f8cc444e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:47:52.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.511+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cbc1a7190 con 0x7f8cbc06c380 2024-03-25T19:47:52.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.538+0000 7f8cb2ffd640 1 -- 172.21.15.18:0/1744683740 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f8cb8052010 con 0x7f8cbc06c380 2024-03-25T19:47:52.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.538+0000 7f8cb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.538+0000 7f8cb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:52.538 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:47:52.538 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:52.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 10 librados: watch_flush enter 2024-03-25T19:47:52.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 10 librados: watch_flush exit 2024-03-25T19:47:52.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-25T19:47:52.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 10 client.4508.objecter successfully canceled tick 2024-03-25T19:47:52.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 >> v1:172.21.15.18:6800/1520000573 conn(0x7f8ca006dfb0 legacy=0x7f8ca0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 10 monclient: shutdown 2024-03-25T19:47:52.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:52.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 >> v1:172.21.15.18:6790/0 conn(0x7f8cbc06c380 legacy=0x7f8cbc1c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 shutdown_connections 2024-03-25T19:47:52.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 >> 172.21.15.18:0/1744683740 conn(0x7f8cbc0b5d10 msgr2=0x7f8cbc0ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:52.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 shutdown_connections 2024-03-25T19:47:52.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 1 -- 172.21.15.18:0/1744683740 wait complete. 2024-03-25T19:47:52.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:52.540+0000 7f8cc444e640 1 librados: shutdown 2024-03-25T19:47:52.554 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:53.221 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:20.751 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.228 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:20.758 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.272 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:20.803 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.276 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:20.806 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.281 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:20.777 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.288 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:20.785 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.289 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:20.819 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.322 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:20.818 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:53.324 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:20.855 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:59.361 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:59.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.479+0000 7f29b6554640 10 monclient: get_monmap_and_config 2024-03-25T19:47:59.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.479+0000 7f29b6554640 10 monclient: build_initial_monmap 2024-03-25T19:47:59.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.479+0000 7f29b6554640 10 monclient: monmap: 2024-03-25T19:47:59.479 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:59.479 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:59.479 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:47:59.480214+0000 2024-03-25T19:47:59.480 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:47:59.480214+0000 2024-03-25T19:47:59.480 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:59.480 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:59.480 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:59.480 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:47:59.480 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:47:59.480 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 1 Processor -- start 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 1 -- start start 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 10 monclient: init 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 10 monclient(hunting): picked mon.noname-a con 0x7f29b00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 10 monclient(hunting): picked mon.noname-b con 0x7f29b013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 10 monclient(hunting): picked mon.noname-c con 0x7f29b01430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.480+0000 7f29b6554640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b0142520 con 0x7f29b00a98f0 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.481+0000 7f29b6554640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b00b28a0 con 0x7f29b01430c0 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.481+0000 7f29b6554640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b00b2a30 con 0x7f29b013f710 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.481+0000 7f29b6554640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.481+0000 7f29b6554640 10 monclient(hunting): authenticate will time out at 5306.607888s 2024-03-25T19:47:59.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.492+0000 7f29b4af1640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f29b01430c0 0x7f29b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:35766/0 (socket says 172.21.15.18:35766) 2024-03-25T19:47:59.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.492+0000 7f29b4af1640 1 -- 172.21.15.18:0/1358953929 learned_addr learned my addr 172.21.15.18:0/1358953929 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:59.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.492+0000 7f29af7fe640 1 --1- 172.21.15.18:0/1358953929 >> v1:172.21.15.18:6790/0 conn(0x7f29b013f710 0x7f29b01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49948/0 (socket says 172.21.15.18:49948) 2024-03-25T19:47:59.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.503+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257858605 0 0) 0x7f29b00b2a30 con 0x7f29b013f710 2024-03-25T19:47:59.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.503+0000 7f29aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.503+0000 7f29aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.503+0000 7f29aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.503+0000 7f29aeffd640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:47:59.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.503+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2998003520 con 0x7f29b013f710 2024-03-25T19:47:59.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.504+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399506410 0 0) 0x7f29b00b28a0 con 0x7f29b01430c0 2024-03-25T19:47:59.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.505+0000 7f29aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.505+0000 7f29aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.505+0000 7f29aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.505+0000 7f29aeffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:47:59.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.505+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b00b2a30 con 0x7f29b01430c0 2024-03-25T19:47:59.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.514+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458545823 0 0) 0x7f29b0142520 con 0x7f29b00a98f0 2024-03-25T19:47:59.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.514+0000 7f29aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.514+0000 7f29aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.514+0000 7f29aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.514+0000 7f29aeffd640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:47:59.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.514+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b00b28a0 con 0x7f29b00a98f0 2024-03-25T19:47:59.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 841675268 0 0) 0x7f2998003520 con 0x7f29b013f710 2024-03-25T19:47:59.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 >> v1:172.21.15.53:6789/0 conn(0x7f29b01430c0 legacy=0x7f29b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 >> v1:172.21.15.18:6789/0 conn(0x7f29b00a98f0 legacy=0x7f29b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:59.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 1 monclient: found mon.noname-b 2024-03-25T19:47:59.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:59.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b00b44c0 con 0x7f29b013f710 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f29a4001690 con 0x7f29b013f710 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29aeffd640 10 monclient: dump: 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:59.524 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:59.525 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:59.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29b6554640 5 monclient: authenticate success, global_id 4511 2024-03-25T19:47:59.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.523+0000 7f29b6554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:59.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.541+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29a40017f0 con 0x7f29b013f710 2024-03-25T19:47:59.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.541+0000 7f29aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:59.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.541+0000 7f29aeffd640 1 -- 172.21.15.18:0/1358953929 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f29a4001c00 con 0x7f29b013f710 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.541+0000 7f29b6554640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.541+0000 7f29b6554640 4 set_mon_vals no callback set 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.541+0000 7f29aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.541+0000 7f29aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.541+0000 7f29aeffd640 10 monclient: dump: 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:59.542 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:59.543 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:59.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.542+0000 7f29b6554640 10 monclient: shutdown 2024-03-25T19:47:59.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.542+0000 7f29b6554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:59.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.542+0000 7f29b6554640 1 -- 172.21.15.18:0/1358953929 >> v1:172.21.15.18:6790/0 conn(0x7f29b013f710 legacy=0x7f29b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.542+0000 7f29b6554640 1 -- 172.21.15.18:0/1358953929 shutdown_connections 2024-03-25T19:47:59.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.542+0000 7f29b6554640 1 -- 172.21.15.18:0/1358953929 >> 172.21.15.18:0/1358953929 conn(0x7f29b00b7ea0 msgr2=0x7f29b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:59.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.543+0000 7f29b6554640 1 -- 172.21.15.18:0/1358953929 shutdown_connections 2024-03-25T19:47:59.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.543+0000 7f29b6554640 1 -- 172.21.15.18:0/1358953929 wait complete. 2024-03-25T19:47:59.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.543+0000 7f29b6554640 10 monclient: build_initial_monmap 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.543+0000 7f29b6554640 10 monclient: monmap: 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.544+0000 7f29b6554640 1 librados: starting msgr at 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.544+0000 7f29b6554640 1 librados: starting objecter 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.544+0000 7f29b6554640 1 Processor -- start 2024-03-25T19:47:59.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.544+0000 7f29b6554640 1 -- start start 2024-03-25T19:47:59.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.544+0000 7f29b6554640 1 librados: setting wanted keys 2024-03-25T19:47:59.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.544+0000 7f29b6554640 1 librados: calling monclient init 2024-03-25T19:47:59.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.544+0000 7f29b6554640 10 monclient: init 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 10 monclient(hunting): picked mon.noname-b con 0x7f29b00a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 10 monclient(hunting): picked mon.noname-a con 0x7f29b013f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 10 monclient(hunting): picked mon.noname-c con 0x7f29b01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b00aba00 con 0x7f29b013f710 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b01d7490 con 0x7f29b00a98f0 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b01d8360 con 0x7f29b01430c0 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:59.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.545+0000 7f29b6554640 10 monclient(hunting): authenticate will time out at 5306.672536s 2024-03-25T19:47:59.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.551+0000 7f29affff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f29b00a98f0 0x7f29b01386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:35836/0 (socket says 172.21.15.18:35836) 2024-03-25T19:47:59.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.551+0000 7f29affff640 1 -- 172.21.15.18:0/1748477082 learned_addr learned my addr 172.21.15.18:0/1748477082 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:47:59.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.552+0000 7f29acff9640 10 client.?.objecter ms_handle_connect 0x7f29b00a98f0 2024-03-25T19:47:59.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.552+0000 7f29acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.553+0000 7f29acff9640 10 client.?.objecter ms_handle_connect 0x7f29b013f710 2024-03-25T19:47:59.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.553+0000 7f29acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.558+0000 7f29acff9640 10 client.?.objecter ms_handle_connect 0x7f29b01430c0 2024-03-25T19:47:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.558+0000 7f29acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.558+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730533954 0 0) 0x7f29b01d7490 con 0x7f29b00a98f0 2024-03-25T19:47:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.559+0000 7f29acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.559+0000 7f29acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.559+0000 7f29acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.559+0000 7f29acff9640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:47:59.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.559+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f299c004eb0 con 0x7f29b00a98f0 2024-03-25T19:47:59.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.563+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617467264 0 0) 0x7f29b00aba00 con 0x7f29b013f710 2024-03-25T19:47:59.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.563+0000 7f29acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.563+0000 7f29acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.563+0000 7f29acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.563+0000 7f29acff9640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:47:59.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.563+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b01d7490 con 0x7f29b013f710 2024-03-25T19:47:59.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.565+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310329187 0 0) 0x7f29b01d8360 con 0x7f29b01430c0 2024-03-25T19:47:59.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.565+0000 7f29acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.565+0000 7f29acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.565+0000 7f29acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.565+0000 7f29acff9640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:47:59.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.565+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b00aba00 con 0x7f29b01430c0 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.567+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741901417 0 0) 0x7f299c004eb0 con 0x7f29b00a98f0 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.567+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29b01d8360 con 0x7f29b00a98f0 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.567+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f29a0001a10 con 0x7f29b00a98f0 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.567+0000 7f29acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.567+0000 7f29acff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.567+0000 7f29acff9640 10 monclient(hunting): dump: 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:59.567 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:59.568 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:59.568 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:59.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.567+0000 7f29acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:59.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.573+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3443767073 0 0) 0x7f29b00aba00 con 0x7f29b01430c0 2024-03-25T19:47:59.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.573+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f299c004eb0 con 0x7f29b01430c0 2024-03-25T19:47:59.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.573+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f29a8001bd0 con 0x7f29b01430c0 2024-03-25T19:47:59.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.573+0000 7f29acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.573+0000 7f29acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.573+0000 7f29acff9640 10 monclient(hunting): dump: 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:59.574 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:59.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.576+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 117878231 0 0) 0x7f29b01d7490 con 0x7f29b013f710 2024-03-25T19:47:59.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.576+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29b00aba00 con 0x7f29b013f710 2024-03-25T19:47:59.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.576+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f29a4001b00 con 0x7f29b013f710 2024-03-25T19:47:59.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.576+0000 7f29acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.576+0000 7f29acff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.576+0000 7f29acff9640 10 monclient(hunting): dump: 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:59.577 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:59.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3864369074 0 0) 0x7f29b01d8360 con 0x7f29b00a98f0 2024-03-25T19:47:59.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 >> v1:172.21.15.18:6790/0 conn(0x7f29b01430c0 legacy=0x7f29b0133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 >> v1:172.21.15.18:6789/0 conn(0x7f29b013f710 legacy=0x7f29b01d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:59.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 1 monclient: found mon.b 2024-03-25T19:47:59.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:59.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b00abc30 con 0x7f29b00a98f0 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29b6554640 5 monclient: authenticate success, global_id 4516 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29b6554640 10 monclient: _renew_subs 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29b6554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.580+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b0137940 con 0x7f29b00a98f0 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.581+0000 7f29b6554640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.581+0000 7f29b6554640 10 monclient: _renew_subs 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.581+0000 7f29b6554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.581+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b0135220 con 0x7f29b00a98f0 2024-03-25T19:47:59.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.581+0000 7f29b6554640 1 librados: init done 2024-03-25T19:47:59.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.582+0000 7f29b6554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:59.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.582+0000 7f29b6554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:59.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.582+0000 7f29b6554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:47:59.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.582+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b00627b0 con 0x7f29b00a98f0 2024-03-25T19:47:59.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.797+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29a0001b70 con 0x7f29b00a98f0 2024-03-25T19:47:59.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.797+0000 7f29acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:59.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.797+0000 7f29ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:59.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.860+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f29a0003a60 con 0x7f29b00a98f0 2024-03-25T19:47:59.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.860+0000 7f29acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.860+0000 7f29acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.860+0000 7f29acff9640 10 monclient: dump: 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:47:59.861 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:47:59.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.861+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f29a0003c40 con 0x7f29b00a98f0 2024-03-25T19:47:59.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.862+0000 7f29af7fe640 10 monclient: get_auth_request con 0x7f299c06e0f0 auth_method 0 2024-03-25T19:47:59.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.862+0000 7f29acff9640 10 client.4516.objecter ms_handle_connect 0x7f299c06e0f0 2024-03-25T19:47:59.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.865+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f29a0054930 con 0x7f29b00a98f0 2024-03-25T19:47:59.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.865+0000 7f29acff9640 10 client.4516.objecter ms_dispatch 0x7f29b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:59.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.865+0000 7f29acff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:59.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.865+0000 7f29acff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:59.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.865+0000 7f29acff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T19:47:59.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.872+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29a0050270 con 0x7f29b00a98f0 2024-03-25T19:47:59.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.872+0000 7f29acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:59.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:47:59.872+0000 7f29acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:00.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.099+0000 7f29b6554640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.099+0000 7f29b6554640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.099+0000 7f29b6554640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:48:00.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.099+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29b01d7830 con 0x7f29b00a98f0 2024-03-25T19:48:00.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.262+0000 7f29acff9640 1 -- 172.21.15.18:0/1748477082 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f29a0052370 con 0x7f29b00a98f0 2024-03-25T19:48:00.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.262+0000 7f29acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.262+0000 7f29acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:00.262 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:48:00.262 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:00.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 10 librados: watch_flush enter 2024-03-25T19:48:00.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 10 librados: watch_flush exit 2024-03-25T19:48:00.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T19:48:00.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 10 client.4516.objecter successfully canceled tick 2024-03-25T19:48:00.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 >> v1:172.21.15.18:6800/1520000573 conn(0x7f299c06e0f0 legacy=0x7f299c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:00.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 10 monclient: shutdown 2024-03-25T19:48:00.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:00.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 >> v1:172.21.15.53:6789/0 conn(0x7f29b00a98f0 legacy=0x7f29b01386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:00.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 shutdown_connections 2024-03-25T19:48:00.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 >> 172.21.15.18:0/1748477082 conn(0x7f29b00b7ea0 msgr2=0x7f29b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:00.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 shutdown_connections 2024-03-25T19:48:00.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 1 -- 172.21.15.18:0/1748477082 wait complete. 2024-03-25T19:48:00.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:00.264+0000 7f29b6554640 1 librados: shutdown 2024-03-25T19:48:00.275 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:07.083 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:07.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.201+0000 7feddbb5d640 10 monclient: get_monmap_and_config 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.201+0000 7feddbb5d640 10 monclient: build_initial_monmap 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.201+0000 7feddbb5d640 10 monclient: monmap: 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:48:07.203698+0000 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:48:07.203698+0000 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:48:07.203 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:07.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.202+0000 7feddbb5d640 1 Processor -- start 2024-03-25T19:48:07.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.202+0000 7feddbb5d640 1 -- start start 2024-03-25T19:48:07.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.202+0000 7feddbb5d640 10 monclient: init 2024-03-25T19:48:07.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fedd40a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fedd413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fedd41430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd4142520 con 0x7fedd41430c0 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd40b28a0 con 0x7fedd413f710 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd40b2a30 con 0x7fedd40a98f0 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.203+0000 7feddbb5d640 10 monclient(hunting): authenticate will time out at 5314.331578s 2024-03-25T19:48:07.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.208+0000 7fedda0fa640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fedd41430c0 0x7fedd414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:40690/0 (socket says 172.21.15.18:40690) 2024-03-25T19:48:07.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.208+0000 7fedda0fa640 1 -- 172.21.15.18:0/2640982268 learned_addr learned my addr 172.21.15.18:0/2640982268 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:07.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.225+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878931646 0 0) 0x7fedd4142520 con 0x7fedd41430c0 2024-03-25T19:48:07.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.225+0000 7fedd88f7640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.225+0000 7fedd88f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.225+0000 7fedd88f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.225+0000 7fedd88f7640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:48:07.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.225+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc0003520 con 0x7fedd41430c0 2024-03-25T19:48:07.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.230+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461899197 0 0) 0x7fedd40b28a0 con 0x7fedd413f710 2024-03-25T19:48:07.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.230+0000 7fedd88f7640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.230+0000 7fedd88f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.230+0000 7fedd88f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.230+0000 7fedd88f7640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:48:07.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.230+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd4142520 con 0x7fedd413f710 2024-03-25T19:48:07.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.232+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031974531 0 0) 0x7fedd40b2a30 con 0x7fedd40a98f0 2024-03-25T19:48:07.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.232+0000 7fedd88f7640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.232+0000 7fedd88f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.232+0000 7fedd88f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.232+0000 7fedd88f7640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:48:07.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.232+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd40b28a0 con 0x7fedd40a98f0 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1487558389 0 0) 0x7fedd4142520 con 0x7fedd413f710 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 >> v1:172.21.15.18:6790/0 conn(0x7fedd40a98f0 legacy=0x7fedd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 >> v1:172.21.15.18:6789/0 conn(0x7fedd41430c0 legacy=0x7fedd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 10 monclient: _finish_hunting 0 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 1 monclient: found mon.noname-c 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40b44c0 con 0x7fedd413f710 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 10 monclient: _finish_auth 0 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 10 monclient: _check_auth_tickets 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fedc4001670 con 0x7fedd413f710 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.239+0000 7feddbb5d640 5 monclient: authenticate success, global_id 4519 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.240+0000 7fedd88f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.240+0000 7fedd88f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.240+0000 7fedd88f7640 10 monclient: dump: 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:07.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.240+0000 7feddbb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:07.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedc40017d0 con 0x7fedd413f710 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7fedd88f7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7fedd88f7640 1 -- 172.21.15.18:0/2640982268 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fedc4001be0 con 0x7fedd413f710 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7feddbb5d640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7feddbb5d640 4 set_mon_vals no callback set 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7fedd88f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7fedd88f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7fedd88f7640 10 monclient: dump: 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:07.263 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:07.264 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:07.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7feddbb5d640 10 monclient: shutdown 2024-03-25T19:48:07.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7feddbb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:07.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7feddbb5d640 1 -- 172.21.15.18:0/2640982268 >> v1:172.21.15.53:6789/0 conn(0x7fedd413f710 legacy=0x7fedd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.262+0000 7feddbb5d640 1 -- 172.21.15.18:0/2640982268 shutdown_connections 2024-03-25T19:48:07.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.263+0000 7feddbb5d640 1 -- 172.21.15.18:0/2640982268 >> 172.21.15.18:0/2640982268 conn(0x7fedd40b7ea0 msgr2=0x7fedd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:07.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.263+0000 7feddbb5d640 1 -- 172.21.15.18:0/2640982268 shutdown_connections 2024-03-25T19:48:07.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.263+0000 7feddbb5d640 1 -- 172.21.15.18:0/2640982268 wait complete. 2024-03-25T19:48:07.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 10 monclient: build_initial_monmap 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 10 monclient: monmap: 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 1 librados: starting msgr at 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 1 librados: starting objecter 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 1 Processor -- start 2024-03-25T19:48:07.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 1 -- start start 2024-03-25T19:48:07.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 1 librados: setting wanted keys 2024-03-25T19:48:07.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 1 librados: calling monclient init 2024-03-25T19:48:07.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.264+0000 7feddbb5d640 10 monclient: init 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fedd40a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fedd413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fedd41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd4136510 con 0x7fedd413f710 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd40abb00 con 0x7fedd40a98f0 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd41da990 con 0x7fedd41430c0 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:07.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.265+0000 7feddbb5d640 10 monclient(hunting): authenticate will time out at 5314.393373s 2024-03-25T19:48:07.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.276+0000 7fedd90f8640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fedd413f710 0x7fedd41383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:40732/0 (socket says 172.21.15.18:40732) 2024-03-25T19:48:07.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.276+0000 7fedd90f8640 1 -- 172.21.15.18:0/454946668 learned_addr learned my addr 172.21.15.18:0/454946668 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:07.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.278+0000 7fedca7fc640 10 client.?.objecter ms_handle_connect 0x7fedd413f710 2024-03-25T19:48:07.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.278+0000 7fedca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.291+0000 7fedca7fc640 10 client.?.objecter ms_handle_connect 0x7fedd41430c0 2024-03-25T19:48:07.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.291+0000 7fedca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.293+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324045482 0 0) 0x7fedd4136510 con 0x7fedd413f710 2024-03-25T19:48:07.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.293+0000 7fedca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.293+0000 7fedca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.293+0000 7fedca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.293+0000 7fedca7fc640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:48:07.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.293+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb8004eb0 con 0x7fedd413f710 2024-03-25T19:48:07.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.293+0000 7fedca7fc640 10 client.?.objecter ms_handle_connect 0x7fedd40a98f0 2024-03-25T19:48:07.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.293+0000 7fedca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.299+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281231639 0 0) 0x7fedd40abb00 con 0x7fedd40a98f0 2024-03-25T19:48:07.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.299+0000 7fedca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.299+0000 7fedca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.299+0000 7fedca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.299+0000 7fedca7fc640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:48:07.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.299+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd4136510 con 0x7fedd40a98f0 2024-03-25T19:48:07.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.306+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472543557 0 0) 0x7fedd41da990 con 0x7fedd41430c0 2024-03-25T19:48:07.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.306+0000 7fedca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.306+0000 7fedca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.306+0000 7fedca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.306+0000 7fedca7fc640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:48:07.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.306+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd40abb00 con 0x7fedd41430c0 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.310+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694330854 0 0) 0x7fedb8004eb0 con 0x7fedd413f710 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.310+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedd41da990 con 0x7fedd413f710 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.310+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fedc4001670 con 0x7fedd413f710 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.310+0000 7fedca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.310+0000 7fedca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.310+0000 7fedca7fc640 10 monclient(hunting): dump: 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:07.311 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:07.312 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:07.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.310+0000 7fedca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.315+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2933680726 0 0) 0x7fedd41da990 con 0x7fedd413f710 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 >> v1:172.21.15.18:6790/0 conn(0x7fedd41430c0 legacy=0x7fedd41e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 >> v1:172.21.15.53:6789/0 conn(0x7fedd40a98f0 legacy=0x7fedd41daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 1 monclient: found mon.a 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd4137820 con 0x7fedd413f710 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:07.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7fedca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 5 monclient: authenticate success, global_id 4554 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 10 monclient: _renew_subs 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd41db5c0 con 0x7fedd413f710 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 10 monclient: _renew_subs 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd4138c90 con 0x7fedd413f710 2024-03-25T19:48:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.316+0000 7feddbb5d640 1 librados: init done 2024-03-25T19:48:07.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.317+0000 7feddbb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.317+0000 7feddbb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.317+0000 7feddbb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:07.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.317+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd40b3ef0 con 0x7fedd413f710 2024-03-25T19:48:07.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.637+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedc40038a0 con 0x7fedd413f710 2024-03-25T19:48:07.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.637+0000 7fedca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:07.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.637+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fedc4003cf0 con 0x7fedd413f710 2024-03-25T19:48:07.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.637+0000 7fedca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.637+0000 7fedca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:07.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.637+0000 7fedca7fc640 10 monclient: dump: 2024-03-25T19:48:07.638 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:07.638 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:07.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.637+0000 7fedcbfff640 4 set_mon_vals no callback set 2024-03-25T19:48:07.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.638+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fedc4003f10 con 0x7fedd413f710 2024-03-25T19:48:07.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.639+0000 7fedd98f9640 10 monclient: get_auth_request con 0x7fedb806ddf0 auth_method 0 2024-03-25T19:48:07.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.639+0000 7fedca7fc640 10 client.4554.objecter ms_handle_connect 0x7fedb806ddf0 2024-03-25T19:48:07.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.644+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fedc40549b0 con 0x7fedd413f710 2024-03-25T19:48:07.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.644+0000 7fedca7fc640 10 client.4554.objecter ms_dispatch 0x7fedd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:07.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.644+0000 7fedca7fc640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:07.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.644+0000 7fedca7fc640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:07.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.644+0000 7fedca7fc640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-25T19:48:07.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.644+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fedc40504c0 con 0x7fedd413f710 2024-03-25T19:48:07.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.644+0000 7fedca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.644+0000 7fedca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:07.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.901+0000 7feddbb5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:07.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.901+0000 7feddbb5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:07.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.901+0000 7feddbb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:07.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.901+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedd414f320 con 0x7fedd413f710 2024-03-25T19:48:07.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.929+0000 7fedca7fc640 1 -- 172.21.15.18:0/454946668 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fedc40523f0 con 0x7fedd413f710 2024-03-25T19:48:07.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.929+0000 7fedca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:07.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.929+0000 7fedca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:07.930 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:48:07.931 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:07.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 10 librados: watch_flush enter 2024-03-25T19:48:07.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 10 librados: watch_flush exit 2024-03-25T19:48:07.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-25T19:48:07.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 10 client.4554.objecter successfully canceled tick 2024-03-25T19:48:07.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 >> v1:172.21.15.18:6800/1520000573 conn(0x7fedb806ddf0 legacy=0x7fedb80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 10 monclient: shutdown 2024-03-25T19:48:07.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:07.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 >> v1:172.21.15.18:6789/0 conn(0x7fedd413f710 legacy=0x7fedd41383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 shutdown_connections 2024-03-25T19:48:07.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 >> 172.21.15.18:0/454946668 conn(0x7fedd40b7ea0 msgr2=0x7fedd4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:07.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 shutdown_connections 2024-03-25T19:48:07.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 1 -- 172.21.15.18:0/454946668 wait complete. 2024-03-25T19:48:07.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:07.932+0000 7feddbb5d640 1 librados: shutdown 2024-03-25T19:48:07.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:13.297 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:40.827 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:13.306 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:40.837 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:13.334 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:40.864 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:13.371 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:40.867 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:13.377 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:40.908 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:13.378 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:40.874 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:13.383 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:40.913 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:13.412 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:07:40.908 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:13.418 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:40.949 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:14.750 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.867+0000 7fa87b86d640 10 monclient: get_monmap_and_config 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.867+0000 7fa87b86d640 10 monclient: build_initial_monmap 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.867+0000 7fa87b86d640 10 monclient: monmap: 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:48:14.869045+0000 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:48:14.869045+0000 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:14.868 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:14.869 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:48:14.869 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:48:14.869 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.867+0000 7fa87b86d640 1 Processor -- start 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 1 -- start start 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 10 monclient: init 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8740b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 10 monclient(hunting): picked mon.noname-c con 0x7fa8740b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 10 monclient(hunting): picked mon.noname-b con 0x7fa87414be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8740af940 con 0x7fa8740b4d60 2024-03-25T19:48:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8740b0a70 con 0x7fa8740b1730 2024-03-25T19:48:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8740b0d50 con 0x7fa87414be90 2024-03-25T19:48:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.868+0000 7fa87b86d640 10 monclient(hunting): authenticate will time out at 5321.996553s 2024-03-25T19:48:14.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.874+0000 7fa878e08640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa8740b1730 0x7fa8740b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:55154/0 (socket says 172.21.15.18:55154) 2024-03-25T19:48:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.874+0000 7fa878e08640 1 -- 172.21.15.18:0/1384784980 learned_addr learned my addr 172.21.15.18:0/1384784980 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:14.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.888+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967766243 0 0) 0x7fa8740b0a70 con 0x7fa8740b1730 2024-03-25T19:48:14.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.888+0000 7fa86bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.888+0000 7fa86bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.888+0000 7fa86bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.888+0000 7fa86bfff640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:48:14.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.888+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa860003520 con 0x7fa8740b1730 2024-03-25T19:48:14.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.903+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151647414 0 0) 0x7fa8740b0d50 con 0x7fa87414be90 2024-03-25T19:48:14.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.903+0000 7fa86bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.903+0000 7fa86bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.903+0000 7fa86bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.903+0000 7fa86bfff640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:48:14.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.903+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8740b0a70 con 0x7fa87414be90 2024-03-25T19:48:14.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.904+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375808371 0 0) 0x7fa8740af940 con 0x7fa8740b4d60 2024-03-25T19:48:14.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.904+0000 7fa86bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.904+0000 7fa86bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.904+0000 7fa86bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.904+0000 7fa86bfff640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:48:14.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.904+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8740b0d50 con 0x7fa8740b4d60 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.911+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636656921 0 0) 0x7fa860003520 con 0x7fa8740b1730 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.911+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 >> v1:172.21.15.18:6790/0 conn(0x7fa87414be90 legacy=0x7fa87414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.911+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 >> v1:172.21.15.18:6789/0 conn(0x7fa8740b4d60 legacy=0x7fa87413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 1 monclient: found mon.noname-c 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8740b1030 con 0x7fa8740b1730 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 10 monclient: _finish_auth 0 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa864001690 con 0x7fa8740b1730 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa86bfff640 10 monclient: dump: 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa87b86d640 5 monclient: authenticate success, global_id 4525 2024-03-25T19:48:14.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.912+0000 7fa87b86d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8640017f0 con 0x7fa8740b1730 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa86bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa86bfff640 1 -- 172.21.15.18:0/1384784980 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa864001c00 con 0x7fa8740b1730 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa87b86d640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa87b86d640 4 set_mon_vals no callback set 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa86bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa86bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa86bfff640 10 monclient: dump: 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:14.933 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:14.934 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:14.934 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:14.934 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:14.934 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:14.934 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.932+0000 7fa87b86d640 10 monclient: shutdown 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.933+0000 7fa87b86d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.933+0000 7fa87b86d640 1 -- 172.21.15.18:0/1384784980 >> v1:172.21.15.53:6789/0 conn(0x7fa8740b1730 legacy=0x7fa8740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.933+0000 7fa87b86d640 1 -- 172.21.15.18:0/1384784980 shutdown_connections 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.933+0000 7fa87b86d640 1 -- 172.21.15.18:0/1384784980 >> 172.21.15.18:0/1384784980 conn(0x7fa8740617a0 msgr2=0x7fa874061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.933+0000 7fa87b86d640 1 -- 172.21.15.18:0/1384784980 shutdown_connections 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.933+0000 7fa87b86d640 1 -- 172.21.15.18:0/1384784980 wait complete. 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 10 monclient: build_initial_monmap 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 10 monclient: monmap: 2024-03-25T19:48:14.936 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 1 librados: starting msgr at 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 1 librados: starting objecter 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 1 Processor -- start 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 1 -- start start 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 1 librados: setting wanted keys 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 1 librados: calling monclient init 2024-03-25T19:48:14.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.934+0000 7fa87b86d640 10 monclient: init 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8740b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 10 monclient(hunting): picked mon.noname-c con 0x7fa8740b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 10 monclient(hunting): picked mon.noname-b con 0x7fa87414be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87413adc0 con 0x7fa8740b1730 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87413b0a0 con 0x7fa87414be90 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87413bbd0 con 0x7fa8740b4d60 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:14.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.935+0000 7fa87b86d640 10 monclient(hunting): authenticate will time out at 5322.063043s 2024-03-25T19:48:14.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.956+0000 7fa879e0a640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa87414be90 0x7fa8741e2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:55200/0 (socket says 172.21.15.18:55200) 2024-03-25T19:48:14.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.956+0000 7fa879e0a640 1 -- 172.21.15.18:0/918456041 learned_addr learned my addr 172.21.15.18:0/918456041 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:14.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.958+0000 7fa869ffb640 10 client.?.objecter ms_handle_connect 0x7fa87414be90 2024-03-25T19:48:14.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.959+0000 7fa869ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:14.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.959+0000 7fa869ffb640 10 client.?.objecter ms_handle_connect 0x7fa8740b1730 2024-03-25T19:48:14.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.959+0000 7fa869ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:14.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.965+0000 7fa869ffb640 10 client.?.objecter ms_handle_connect 0x7fa8740b4d60 2024-03-25T19:48:14.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.965+0000 7fa869ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:14.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.976+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127435669 0 0) 0x7fa87413bbd0 con 0x7fa8740b4d60 2024-03-25T19:48:14.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.976+0000 7fa869ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.976+0000 7fa869ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.976+0000 7fa869ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.976+0000 7fa869ffb640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:48:14.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.976+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa858004eb0 con 0x7fa8740b4d60 2024-03-25T19:48:14.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356412350 0 0) 0x7fa87413adc0 con 0x7fa8740b1730 2024-03-25T19:48:14.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87413bbd0 con 0x7fa8740b1730 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954433926 0 0) 0x7fa87413b0a0 con 0x7fa87414be90 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:48:14.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.978+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87413adc0 con 0x7fa87414be90 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3743363991 0 0) 0x7fa858004eb0 con 0x7fa8740b4d60 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa87413b0a0 con 0x7fa8740b4d60 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa864001b00 con 0x7fa8740b4d60 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 10 monclient(hunting): dump: 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:15.001 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905363295 0 0) 0x7fa87413adc0 con 0x7fa87414be90 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa858004eb0 con 0x7fa87414be90 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa86c001bd0 con 0x7fa87414be90 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:14.999+0000 7fa869ffb640 10 monclient(hunting): dump: 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:15.002 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:15.003 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:15.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1763642607 0 0) 0x7fa87413b0a0 con 0x7fa8740b4d60 2024-03-25T19:48:15.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 >> v1:172.21.15.53:6789/0 conn(0x7fa87414be90 legacy=0x7fa8741e2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 >> v1:172.21.15.18:6789/0 conn(0x7fa8740b1730 legacy=0x7fa874139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 1 monclient: found mon.c 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87413bd60 con 0x7fa8740b4d60 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa869ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 10 monclient: _renew_subs 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa874143150 con 0x7fa8740b4d60 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 10 monclient: _renew_subs 2024-03-25T19:48:15.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:15.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa874139f80 con 0x7fa8740b4d60 2024-03-25T19:48:15.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.020+0000 7fa87b86d640 1 librados: init done 2024-03-25T19:48:15.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.021+0000 7fa87b86d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:15.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.021+0000 7fa87b86d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:15.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.021+0000 7fa87b86d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:15.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.021+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8740b58b0 con 0x7fa8740b4d60 2024-03-25T19:48:15.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.342+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa864001220 con 0x7fa8740b4d60 2024-03-25T19:48:15.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.342+0000 7fa869ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.342+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fa864004c30 con 0x7fa8740b4d60 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.342+0000 7fa869ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.342+0000 7fa869ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.342+0000 7fa869ffb640 10 monclient: dump: 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:15.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.342+0000 7fa86b7fe640 4 set_mon_vals no callback set 2024-03-25T19:48:15.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.343+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fa86401ac00 con 0x7fa8740b4d60 2024-03-25T19:48:15.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.344+0000 7fa879609640 10 monclient: get_auth_request con 0x7fa85806df60 auth_method 0 2024-03-25T19:48:15.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.344+0000 7fa869ffb640 10 client.4526.objecter ms_handle_connect 0x7fa85806df60 2024-03-25T19:48:15.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.349+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fa864054810 con 0x7fa8740b4d60 2024-03-25T19:48:15.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.349+0000 7fa869ffb640 10 client.4526.objecter ms_dispatch 0x7fa874000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:15.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.349+0000 7fa869ffb640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:15.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.349+0000 7fa869ffb640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:15.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.349+0000 7fa869ffb640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-25T19:48:15.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.349+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa864050150 con 0x7fa8740b4d60 2024-03-25T19:48:15.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.349+0000 7fa869ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:15.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.349+0000 7fa869ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:15.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.608+0000 7fa87b86d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.608+0000 7fa87b86d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.608+0000 7fa87b86d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:15.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.608+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8740b5f40 con 0x7fa8740b4d60 2024-03-25T19:48:15.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.649+0000 7fa869ffb640 1 -- 172.21.15.18:0/918456041 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fa864052250 con 0x7fa8740b4d60 2024-03-25T19:48:15.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.649+0000 7fa869ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.649+0000 7fa869ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:15.651 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:48:15.651 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:15.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.651+0000 7fa87b86d640 10 librados: watch_flush enter 2024-03-25T19:48:15.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.651+0000 7fa87b86d640 10 librados: watch_flush exit 2024-03-25T19:48:15.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-25T19:48:15.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 10 client.4526.objecter successfully canceled tick 2024-03-25T19:48:15.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 >> v1:172.21.15.18:6800/1520000573 conn(0x7fa85806df60 legacy=0x7fa858070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 10 monclient: shutdown 2024-03-25T19:48:15.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:15.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 >> v1:172.21.15.18:6790/0 conn(0x7fa8740b4d60 legacy=0x7fa87413b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 shutdown_connections 2024-03-25T19:48:15.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 >> 172.21.15.18:0/918456041 conn(0x7fa8740617a0 msgr2=0x7fa8740b6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:15.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 shutdown_connections 2024-03-25T19:48:15.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 1 -- 172.21.15.18:0/918456041 wait complete. 2024-03-25T19:48:15.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:15.652+0000 7fa87b86d640 1 librados: shutdown 2024-03-25T19:48:15.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:22.472 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:22.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.587+0000 7ff6db5a3640 10 monclient: get_monmap_and_config 2024-03-25T19:48:22.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.587+0000 7ff6db5a3640 10 monclient: build_initial_monmap 2024-03-25T19:48:22.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.587+0000 7ff6db5a3640 10 monclient: monmap: 2024-03-25T19:48:22.588 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:22.588 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:22.588 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:48:22.589398+0000 2024-03-25T19:48:22.588 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:48:22.589398+0000 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.588+0000 7ff6db5a3640 1 Processor -- start 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.588+0000 7ff6db5a3640 1 -- start start 2024-03-25T19:48:22.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.588+0000 7ff6db5a3640 10 monclient: init 2024-03-25T19:48:22.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:22.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:22.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d40a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:22.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:22.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:22.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d4142520 con 0x7ff6d40a98f0 2024-03-25T19:48:22.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40b28a0 con 0x7ff6d413f710 2024-03-25T19:48:22.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40b2a30 con 0x7ff6d41430c0 2024-03-25T19:48:22.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:22.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.589+0000 7ff6db5a3640 10 monclient(hunting): authenticate will time out at 5329.717127s 2024-03-25T19:48:22.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.600+0000 7ff6d9b40640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ff6d41430c0 0x7ff6d414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35522/0 (socket says 172.21.15.18:35522) 2024-03-25T19:48:22.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.600+0000 7ff6d9b40640 1 -- 172.21.15.18:0/175308558 learned_addr learned my addr 172.21.15.18:0/175308558 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:22.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.613+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088126327 0 0) 0x7ff6d4142520 con 0x7ff6d40a98f0 2024-03-25T19:48:22.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.613+0000 7ff6cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.613+0000 7ff6cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.613+0000 7ff6cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.613+0000 7ff6cbfff640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:48:22.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.613+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c0003520 con 0x7ff6d40a98f0 2024-03-25T19:48:22.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.618+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162771129 0 0) 0x7ff6d40b2a30 con 0x7ff6d41430c0 2024-03-25T19:48:22.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.618+0000 7ff6cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.618+0000 7ff6cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.618+0000 7ff6cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.618+0000 7ff6cbfff640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:48:22.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.618+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d4142520 con 0x7ff6d41430c0 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.620+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1597477042 0 0) 0x7ff6c0003520 con 0x7ff6d40a98f0 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 >> v1:172.21.15.18:6790/0 conn(0x7ff6d41430c0 legacy=0x7ff6d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 >> v1:172.21.15.53:6789/0 conn(0x7ff6d413f710 legacy=0x7ff6d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 1 monclient: found mon.noname-a 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d40b44c0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 10 monclient: _finish_auth 0 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:22.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff6d0001690 con 0x7ff6d40a98f0 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6cbfff640 10 monclient: dump: 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6db5a3640 5 monclient: authenticate success, global_id 4563 2024-03-25T19:48:22.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.621+0000 7ff6db5a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.632+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d00017f0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.632+0000 7ff6cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.632+0000 7ff6cbfff640 1 -- 172.21.15.18:0/175308558 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff6d0001c00 con 0x7ff6d40a98f0 2024-03-25T19:48:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.632+0000 7ff6cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.632+0000 7ff6cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.632+0000 7ff6cbfff640 10 monclient: dump: 2024-03-25T19:48:22.633 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:22.633 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.632+0000 7ff6db5a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:22.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.632+0000 7ff6db5a3640 4 set_mon_vals no callback set 2024-03-25T19:48:22.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.633+0000 7ff6db5a3640 10 monclient: shutdown 2024-03-25T19:48:22.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.633+0000 7ff6db5a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:22.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.633+0000 7ff6db5a3640 1 -- 172.21.15.18:0/175308558 >> v1:172.21.15.18:6789/0 conn(0x7ff6d40a98f0 legacy=0x7ff6d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.633+0000 7ff6db5a3640 1 -- 172.21.15.18:0/175308558 shutdown_connections 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.633+0000 7ff6db5a3640 1 -- 172.21.15.18:0/175308558 >> 172.21.15.18:0/175308558 conn(0x7ff6d40b7ea0 msgr2=0x7ff6d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.633+0000 7ff6db5a3640 1 -- 172.21.15.18:0/175308558 shutdown_connections 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.633+0000 7ff6db5a3640 1 -- 172.21.15.18:0/175308558 wait complete. 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.634+0000 7ff6db5a3640 10 monclient: build_initial_monmap 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.634+0000 7ff6db5a3640 10 monclient: monmap: 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:48:22.637 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.634+0000 7ff6db5a3640 1 librados: starting msgr at 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.634+0000 7ff6db5a3640 1 librados: starting objecter 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 1 Processor -- start 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 1 -- start start 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 1 librados: setting wanted keys 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 1 librados: calling monclient init 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 10 monclient: init 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d40a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d413f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d41430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40b1750 con 0x7ff6d41430c0 2024-03-25T19:48:22.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d41399e0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d40ab950 con 0x7ff6d413f710 2024-03-25T19:48:22.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:22.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.635+0000 7ff6db5a3640 10 monclient(hunting): authenticate will time out at 5329.763522s 2024-03-25T19:48:22.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.642+0000 7ff6d933f640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff6d40a98f0 0x7ff6d40ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:46588/0 (socket says 172.21.15.18:46588) 2024-03-25T19:48:22.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.642+0000 7ff6d933f640 1 -- 172.21.15.18:0/4165959428 learned_addr learned my addr 172.21.15.18:0/4165959428 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:22.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.642+0000 7ff6c9ffb640 10 client.?.objecter ms_handle_connect 0x7ff6d40a98f0 2024-03-25T19:48:22.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.642+0000 7ff6c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.647+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295385931 0 0) 0x7ff6d41399e0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.647+0000 7ff6c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.647+0000 7ff6c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.647+0000 7ff6c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.647+0000 7ff6c9ffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:48:22.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.647+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b8004eb0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.647+0000 7ff6c9ffb640 10 client.?.objecter ms_handle_connect 0x7ff6d413f710 2024-03-25T19:48:22.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.647+0000 7ff6c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.661+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250179742 0 0) 0x7ff6b8004eb0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.661+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6d41399e0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.661+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff6d0001c00 con 0x7ff6d40a98f0 2024-03-25T19:48:22.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.661+0000 7ff6c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.661+0000 7ff6c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.661+0000 7ff6c9ffb640 10 monclient(hunting): dump: 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:22.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.661+0000 7ff6c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:22.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.662+0000 7ff6c9ffb640 10 client.?.objecter ms_handle_connect 0x7ff6d41430c0 2024-03-25T19:48:22.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.662+0000 7ff6c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.672+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990507834 0 0) 0x7ff6d40b1750 con 0x7ff6d41430c0 2024-03-25T19:48:22.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.672+0000 7ff6c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.672+0000 7ff6c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.672+0000 7ff6c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.672+0000 7ff6c9ffb640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:48:22.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.672+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b8004eb0 con 0x7ff6d41430c0 2024-03-25T19:48:22.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4027490556 0 0) 0x7ff6d41399e0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 >> v1:172.21.15.18:6790/0 conn(0x7ff6d413f710 legacy=0x7ff6d4136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 >> v1:172.21.15.18:6789/0 conn(0x7ff6d41430c0 legacy=0x7ff6d41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:22.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 1 monclient: found mon.b 2024-03-25T19:48:22.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:22.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:48:22.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d40abc30 con 0x7ff6d40a98f0 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 5 monclient: authenticate success, global_id 4534 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 10 monclient: _renew_subs 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d4138ea0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 10 monclient: _renew_subs 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d41387c0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.674+0000 7ff6db5a3640 1 librados: init done 2024-03-25T19:48:22.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.675+0000 7ff6db5a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:22.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.675+0000 7ff6db5a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:22.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.675+0000 7ff6db5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:48:22.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.675+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d40627b0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.953+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d00038a0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.953+0000 7ff6c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.953+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff6d0003c30 con 0x7ff6d40a98f0 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.953+0000 7ff6c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.953+0000 7ff6c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.953+0000 7ff6c9ffb640 10 monclient: dump: 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:22.955 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:22.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.953+0000 7ff6cb7fe640 4 set_mon_vals no callback set 2024-03-25T19:48:22.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.954+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ff6d0003e50 con 0x7ff6d40a98f0 2024-03-25T19:48:22.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.955+0000 7ff6d8b3e640 10 monclient: get_auth_request con 0x7ff6b806ddb0 auth_method 0 2024-03-25T19:48:22.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.955+0000 7ff6c9ffb640 10 client.4534.objecter ms_handle_connect 0x7ff6b806ddb0 2024-03-25T19:48:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.961+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7ff6d00549c0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.961+0000 7ff6c9ffb640 10 client.4534.objecter ms_dispatch 0x7ff6d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.961+0000 7ff6c9ffb640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.961+0000 7ff6c9ffb640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:22.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.961+0000 7ff6c9ffb640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T19:48:22.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.962+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6d00504d0 con 0x7ff6d40a98f0 2024-03-25T19:48:22.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.962+0000 7ff6c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:22.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:22.962+0000 7ff6c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:23.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.191+0000 7ff6db5a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.191+0000 7ff6db5a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.191+0000 7ff6db5a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:48:23.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.191+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d4139030 con 0x7ff6d40a98f0 2024-03-25T19:48:23.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.226+0000 7ff6c9ffb640 1 -- 172.21.15.18:0/4165959428 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7ff6d0052400 con 0x7ff6d40a98f0 2024-03-25T19:48:23.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.226+0000 7ff6c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.226+0000 7ff6c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:23.227 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:48:23.227 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:23.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 10 librados: watch_flush enter 2024-03-25T19:48:23.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 10 librados: watch_flush exit 2024-03-25T19:48:23.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T19:48:23.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 10 client.4534.objecter successfully canceled tick 2024-03-25T19:48:23.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 >> v1:172.21.15.18:6800/1520000573 conn(0x7ff6b806ddb0 legacy=0x7ff6b8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:23.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 10 monclient: shutdown 2024-03-25T19:48:23.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:23.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 >> v1:172.21.15.53:6789/0 conn(0x7ff6d40a98f0 legacy=0x7ff6d40ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:23.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 shutdown_connections 2024-03-25T19:48:23.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 >> 172.21.15.18:0/4165959428 conn(0x7ff6d40b7ea0 msgr2=0x7ff6d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:23.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 shutdown_connections 2024-03-25T19:48:23.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 1 -- 172.21.15.18:0/4165959428 wait complete. 2024-03-25T19:48:23.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:23.229+0000 7ff6db5a3640 1 librados: shutdown 2024-03-25T19:48:23.243 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:30.050 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:30.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.154+0000 7feca506b640 10 monclient: get_monmap_and_config 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.154+0000 7feca506b640 10 monclient: build_initial_monmap 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.154+0000 7feca506b640 10 monclient: monmap: 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:48:30.155787+0000 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:48:30.155787+0000 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:30.155 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:48:30.156 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:48:30.156 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.155+0000 7feca506b640 1 Processor -- start 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.155+0000 7feca506b640 1 -- start start 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.155+0000 7feca506b640 10 monclient: init 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 10 monclient(hunting): picked mon.noname-c con 0x7feca00a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 10 monclient(hunting): picked mon.noname-a con 0x7feca013f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 10 monclient(hunting): picked mon.noname-b con 0x7feca01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca0142520 con 0x7feca013f710 2024-03-25T19:48:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca00b28a0 con 0x7feca00a98f0 2024-03-25T19:48:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca00b2a30 con 0x7feca01430c0 2024-03-25T19:48:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.156+0000 7feca506b640 10 monclient(hunting): authenticate will time out at 5337.283212s 2024-03-25T19:48:30.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.165+0000 7fec9ed9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7feca00a98f0 0x7feca00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:36590/0 (socket says 172.21.15.18:36590) 2024-03-25T19:48:30.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.165+0000 7fec9ed9d640 1 -- 172.21.15.18:0/221009460 learned_addr learned my addr 172.21.15.18:0/221009460 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:30.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.181+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591914239 0 0) 0x7feca00b28a0 con 0x7feca00a98f0 2024-03-25T19:48:30.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.181+0000 7fec9dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.181+0000 7fec9dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.181+0000 7fec9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.181+0000 7fec9dd9b640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:48:30.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.181+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec88003520 con 0x7feca00a98f0 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.190+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182452724 0 0) 0x7fec88003520 con 0x7feca00a98f0 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.190+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 >> v1:172.21.15.18:6790/0 conn(0x7feca01430c0 legacy=0x7feca014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.190+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 >> v1:172.21.15.18:6789/0 conn(0x7feca013f710 legacy=0x7feca01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 1 monclient: found mon.noname-c 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca00b44c0 con 0x7feca00a98f0 2024-03-25T19:48:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fec90001690 con 0x7feca00a98f0 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7fec9dd9b640 10 monclient: dump: 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:30.192 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:30.193 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:30.193 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:30.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7feca506b640 5 monclient: authenticate success, global_id 4537 2024-03-25T19:48:30.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.191+0000 7feca506b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:30.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.195+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec90001860 con 0x7feca00a98f0 2024-03-25T19:48:30.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.195+0000 7fec9dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:30.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.195+0000 7fec9dd9b640 1 -- 172.21.15.18:0/221009460 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fec90001cb0 con 0x7feca00a98f0 2024-03-25T19:48:30.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.195+0000 7feca506b640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:30.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.195+0000 7feca506b640 4 set_mon_vals no callback set 2024-03-25T19:48:30.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.195+0000 7fec9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.195+0000 7fec9dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:30.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.195+0000 7fec9dd9b640 10 monclient: dump: 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:30.196 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.196+0000 7feca506b640 10 monclient: shutdown 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.196+0000 7feca506b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.196+0000 7feca506b640 1 -- 172.21.15.18:0/221009460 >> v1:172.21.15.53:6789/0 conn(0x7feca00a98f0 legacy=0x7feca00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.196+0000 7feca506b640 1 -- 172.21.15.18:0/221009460 shutdown_connections 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.196+0000 7feca506b640 1 -- 172.21.15.18:0/221009460 >> 172.21.15.18:0/221009460 conn(0x7feca00b7ea0 msgr2=0x7feca00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.196+0000 7feca506b640 1 -- 172.21.15.18:0/221009460 shutdown_connections 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.196+0000 7feca506b640 1 -- 172.21.15.18:0/221009460 wait complete. 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.197+0000 7feca506b640 10 monclient: build_initial_monmap 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.197+0000 7feca506b640 10 monclient: monmap: 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:48:30.198 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.197+0000 7feca506b640 1 librados: starting msgr at 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.197+0000 7feca506b640 1 librados: starting objecter 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 1 Processor -- start 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 1 -- start start 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 1 librados: setting wanted keys 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 1 librados: calling monclient init 2024-03-25T19:48:30.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 10 monclient: init 2024-03-25T19:48:30.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 10 monclient(hunting): picked mon.noname-b con 0x7feca00a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 10 monclient(hunting): picked mon.noname-a con 0x7feca013f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 10 monclient(hunting): picked mon.noname-c con 0x7feca01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca006fde0 con 0x7feca013f710 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca00700c0 con 0x7feca00a98f0 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca006e020 con 0x7feca01430c0 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:30.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.198+0000 7feca506b640 10 monclient(hunting): authenticate will time out at 5337.325534s 2024-03-25T19:48:30.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.204+0000 7fec9f59e640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7feca01430c0 0x7feca01df200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46546/0 (socket says 172.21.15.18:46546) 2024-03-25T19:48:30.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.204+0000 7fec9f59e640 1 -- 172.21.15.18:0/925570634 learned_addr learned my addr 172.21.15.18:0/925570634 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:30.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.205+0000 7fec877fe640 10 client.?.objecter ms_handle_connect 0x7feca01430c0 2024-03-25T19:48:30.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.205+0000 7fec877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.216+0000 7fec877fe640 10 client.?.objecter ms_handle_connect 0x7feca00a98f0 2024-03-25T19:48:30.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.216+0000 7fec877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.221+0000 7fec877fe640 10 client.?.objecter ms_handle_connect 0x7feca013f710 2024-03-25T19:48:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.221+0000 7fec877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.229+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986589965 0 0) 0x7feca00700c0 con 0x7feca00a98f0 2024-03-25T19:48:30.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.229+0000 7fec877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.229+0000 7fec877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.229+0000 7fec877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.229+0000 7fec877fe640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:48:30.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.229+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c004eb0 con 0x7feca00a98f0 2024-03-25T19:48:30.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.232+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963673055 0 0) 0x7feca006fde0 con 0x7feca013f710 2024-03-25T19:48:30.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.232+0000 7fec877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.232+0000 7fec877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.232+0000 7fec877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.232+0000 7fec877fe640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:48:30.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.232+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca00700c0 con 0x7feca013f710 2024-03-25T19:48:30.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.233+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953638749 0 0) 0x7feca006e020 con 0x7feca01430c0 2024-03-25T19:48:30.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.233+0000 7fec877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.233+0000 7fec877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.233+0000 7fec877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.233+0000 7fec877fe640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:48:30.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.233+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca006fde0 con 0x7feca01430c0 2024-03-25T19:48:30.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.248+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 156316705 0 0) 0x7feca00700c0 con 0x7feca013f710 2024-03-25T19:48:30.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.248+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca006e020 con 0x7feca013f710 2024-03-25T19:48:30.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.248+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fec94001810 con 0x7feca013f710 2024-03-25T19:48:30.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.248+0000 7fec877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.248+0000 7fec877fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:30.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.248+0000 7fec877fe640 10 monclient(hunting): dump: 2024-03-25T19:48:30.248 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:30.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.248+0000 7fec877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:30.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.251+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388536128 0 0) 0x7fec8c004eb0 con 0x7feca00a98f0 2024-03-25T19:48:30.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.251+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca00700c0 con 0x7feca00a98f0 2024-03-25T19:48:30.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.251+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fec90001b70 con 0x7feca00a98f0 2024-03-25T19:48:30.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.251+0000 7fec877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.251+0000 7fec877fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:30.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.251+0000 7fec877fe640 10 monclient(hunting): dump: 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:30.252 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.257+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978582839 0 0) 0x7feca006fde0 con 0x7feca01430c0 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.257+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec8c004eb0 con 0x7feca01430c0 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.257+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fec98001bd0 con 0x7feca01430c0 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.257+0000 7fec877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.257+0000 7fec877fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.257+0000 7fec877fe640 10 monclient(hunting): dump: 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.257 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:30.258 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:30.258 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:30.258 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:30.258 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2600831468 0 0) 0x7feca006e020 con 0x7feca013f710 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 >> v1:172.21.15.18:6790/0 conn(0x7feca01430c0 legacy=0x7feca01df200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 >> v1:172.21.15.53:6789/0 conn(0x7feca00a98f0 legacy=0x7feca00707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 1 monclient: found mon.a 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca0070ed0 con 0x7feca013f710 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7fec877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 5 monclient: authenticate success, global_id 4569 2024-03-25T19:48:30.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 10 monclient: _renew_subs 2024-03-25T19:48:30.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:30.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca006e5a0 con 0x7feca013f710 2024-03-25T19:48:30.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:30.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 10 monclient: _renew_subs 2024-03-25T19:48:30.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:30.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feca006e8f0 con 0x7feca013f710 2024-03-25T19:48:30.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.260+0000 7feca506b640 1 librados: init done 2024-03-25T19:48:30.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.262+0000 7feca506b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.262+0000 7feca506b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.262+0000 7feca506b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:30.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.262+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca00b3ef0 con 0x7feca013f710 2024-03-25T19:48:30.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.608+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec94001d00 con 0x7feca013f710 2024-03-25T19:48:30.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.608+0000 7fec877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:30.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.608+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fec94003790 con 0x7feca013f710 2024-03-25T19:48:30.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.608+0000 7fec877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.608+0000 7fec877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.608+0000 7fec877fe640 10 monclient: dump: 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:30.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.608+0000 7fec9d59a640 4 set_mon_vals no callback set 2024-03-25T19:48:30.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.609+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fec94019760 con 0x7feca013f710 2024-03-25T19:48:30.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.609+0000 7fec9ed9d640 10 monclient: get_auth_request con 0x7fec8c06e060 auth_method 0 2024-03-25T19:48:30.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.609+0000 7fec877fe640 10 client.4569.objecter ms_handle_connect 0x7fec8c06e060 2024-03-25T19:48:30.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.618+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fec94053500 con 0x7feca013f710 2024-03-25T19:48:30.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.618+0000 7fec877fe640 10 client.4569.objecter ms_dispatch 0x7feca0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:30.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.618+0000 7fec877fe640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:30.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.618+0000 7fec877fe640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:30.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.618+0000 7fec877fe640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-25T19:48:30.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.618+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec9404ee40 con 0x7feca013f710 2024-03-25T19:48:30.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.618+0000 7fec877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.618+0000 7fec877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:30.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.879+0000 7feca506b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:30.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.879+0000 7feca506b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:30.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.879+0000 7feca506b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:30.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.879+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feca006f390 con 0x7feca013f710 2024-03-25T19:48:30.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.908+0000 7fec877fe640 1 -- 172.21.15.18:0/925570634 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fec94050f40 con 0x7feca013f710 2024-03-25T19:48:30.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.908+0000 7fec877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:30.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.908+0000 7fec877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:30.908 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:48:30.908 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:30.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 10 librados: watch_flush enter 2024-03-25T19:48:30.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 10 librados: watch_flush exit 2024-03-25T19:48:30.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-25T19:48:30.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 10 client.4569.objecter successfully canceled tick 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 >> v1:172.21.15.18:6800/1520000573 conn(0x7fec8c06e060 legacy=0x7fec8c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 10 monclient: shutdown 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 >> v1:172.21.15.18:6789/0 conn(0x7feca013f710 legacy=0x7feca014cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 shutdown_connections 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 >> 172.21.15.18:0/925570634 conn(0x7feca00b7ea0 msgr2=0x7feca0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 shutdown_connections 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 1 -- 172.21.15.18:0/925570634 wait complete. 2024-03-25T19:48:30.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:30.910+0000 7feca506b640 1 librados: shutdown 2024-03-25T19:48:30.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:33.377 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:00.908 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:33.386 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:00.917 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:33.425 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:00.955 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:33.447 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:00.944 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:33.450 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:00.981 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:33.453 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:00.950 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:33.460 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:00.991 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:33.483 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:00.979 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:33.503 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:01.034 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:37.728 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:37.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.849+0000 7f9760bc4640 10 monclient: get_monmap_and_config 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.849+0000 7f9760bc4640 10 monclient: build_initial_monmap 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.849+0000 7f9760bc4640 10 monclient: monmap: 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:48:37.850660+0000 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:48:37.850660+0000 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:48:37.850 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.850+0000 7f9760bc4640 1 Processor -- start 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.850+0000 7f9760bc4640 1 -- start start 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.850+0000 7f9760bc4640 10 monclient: init 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 10 monclient(hunting): picked mon.noname-c con 0x7f975c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 10 monclient(hunting): picked mon.noname-a con 0x7f975c13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 10 monclient(hunting): picked mon.noname-b con 0x7f975c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c142520 con 0x7f975c13f710 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0b28a0 con 0x7f975c0a98f0 2024-03-25T19:48:37.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0b2a30 con 0x7f975c1430c0 2024-03-25T19:48:37.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:37.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.851+0000 7f9760bc4640 10 monclient(hunting): authenticate will time out at 5344.978711s 2024-03-25T19:48:37.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.856+0000 7f975a59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f975c0a98f0 0x7f975c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:46282/0 (socket says 172.21.15.18:46282) 2024-03-25T19:48:37.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.856+0000 7f975a59c640 1 -- 172.21.15.18:0/1612970997 learned_addr learned my addr 172.21.15.18:0/1612970997 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:37.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.933+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23429553 0 0) 0x7f975c142520 con 0x7f975c13f710 2024-03-25T19:48:37.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.933+0000 7f975959a640 10 monclient(hunting): _negotiate 2024-03-25T19:48:37.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.933+0000 7f975959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:37.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.933+0000 7f975959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:37.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.933+0000 7f975959a640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:48:37.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.933+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9744003520 con 0x7f975c13f710 2024-03-25T19:48:37.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 93615526 0 0) 0x7f9744003520 con 0x7f975c13f710 2024-03-25T19:48:37.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 >> v1:172.21.15.18:6790/0 conn(0x7f975c1430c0 legacy=0x7f975c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 >> v1:172.21.15.53:6789/0 conn(0x7f975c0a98f0 legacy=0x7f975c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 10 monclient: _finish_hunting 0 2024-03-25T19:48:37.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 1 monclient: found mon.noname-a 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c0b44c0 con 0x7f975c13f710 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 10 monclient: _finish_auth 0 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 10 monclient: _check_auth_tickets 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9750001690 con 0x7f975c13f710 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f9760bc4640 5 monclient: authenticate success, global_id 4572 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f9760bc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f975959a640 10 monclient: dump: 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:37.952 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:37.953 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:37.953 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:37.953 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:37.953 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:37.953 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:37.953 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:37.953 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:37.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.951+0000 7f9760bc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97500017f0 con 0x7f975c13f710 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f975959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f975959a640 1 -- 172.21.15.18:0/1612970997 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9750001c00 con 0x7f975c13f710 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f9760bc4640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f9760bc4640 4 set_mon_vals no callback set 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f975959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f975959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f975959a640 10 monclient: dump: 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:37.968 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:37.969 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:37.969 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:37.969 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:37.969 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:37.969 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:37.969 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f9760bc4640 10 monclient: shutdown 2024-03-25T19:48:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f9760bc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.968+0000 7f9760bc4640 1 -- 172.21.15.18:0/1612970997 >> v1:172.21.15.18:6789/0 conn(0x7f975c13f710 legacy=0x7f975c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.969+0000 7f9760bc4640 1 -- 172.21.15.18:0/1612970997 shutdown_connections 2024-03-25T19:48:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.969+0000 7f9760bc4640 1 -- 172.21.15.18:0/1612970997 >> 172.21.15.18:0/1612970997 conn(0x7f975c0b7ea0 msgr2=0x7f975c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.969+0000 7f9760bc4640 1 -- 172.21.15.18:0/1612970997 shutdown_connections 2024-03-25T19:48:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.969+0000 7f9760bc4640 1 -- 172.21.15.18:0/1612970997 wait complete. 2024-03-25T19:48:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.969+0000 7f9760bc4640 10 monclient: build_initial_monmap 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.969+0000 7f9760bc4640 10 monclient: monmap: 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 librados: starting msgr at 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 librados: starting objecter 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 Processor -- start 2024-03-25T19:48:37.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 -- start start 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 librados: setting wanted keys 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 librados: calling monclient init 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 10 monclient: init 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 10 monclient(hunting): picked mon.noname-c con 0x7f975c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 10 monclient(hunting): picked mon.noname-b con 0x7f975c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 10 monclient(hunting): picked mon.noname-a con 0x7f975c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c0abd70 con 0x7f975c1430c0 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c1d7710 con 0x7f975c13f710 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.970+0000 7f9760bc4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f975c1d8340 con 0x7f975c0a98f0 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.971+0000 7f9760bc4640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:37.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.971+0000 7f9760bc4640 10 monclient(hunting): authenticate will time out at 5345.097821s 2024-03-25T19:48:37.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.978+0000 7f975ad9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f975c1430c0 0x7f975c1d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:59218/0 (socket says 172.21.15.18:59218) 2024-03-25T19:48:37.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.978+0000 7f975ad9d640 1 -- 172.21.15.18:0/2793778444 learned_addr learned my addr 172.21.15.18:0/2793778444 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:37.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.979+0000 7f974affd640 10 client.?.objecter ms_handle_connect 0x7f975c1430c0 2024-03-25T19:48:37.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.979+0000 7f974affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:37.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.982+0000 7f974affd640 10 client.?.objecter ms_handle_connect 0x7f975c0a98f0 2024-03-25T19:48:37.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.982+0000 7f974affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:37.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.991+0000 7f974affd640 10 client.?.objecter ms_handle_connect 0x7f975c13f710 2024-03-25T19:48:37.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.991+0000 7f974affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:37.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.991+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162205786 0 0) 0x7f975c0abd70 con 0x7f975c1430c0 2024-03-25T19:48:37.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.992+0000 7f974affd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:37.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.992+0000 7f974affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:37.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.992+0000 7f974affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:37.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.992+0000 7f974affd640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:48:37.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:37.992+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c004eb0 con 0x7f975c1430c0 2024-03-25T19:48:38.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.007+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419634261 0 0) 0x7f975c1d8340 con 0x7f975c0a98f0 2024-03-25T19:48:38.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.007+0000 7f974affd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:38.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.007+0000 7f974affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:38.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.007+0000 7f974affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:38.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.007+0000 7f974affd640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:48:38.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.007+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c0abd70 con 0x7f975c0a98f0 2024-03-25T19:48:38.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.011+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2244308493 0 0) 0x7f973c004eb0 con 0x7f975c1430c0 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.011+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f975c1d8340 con 0x7f975c1430c0 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.011+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9754001c30 con 0x7f975c1430c0 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.011+0000 7f974affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.011+0000 7f974affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.011+0000 7f974affd640 10 monclient(hunting): dump: 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:38.011 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:38.012 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:38.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.011+0000 7f974affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:38.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.018+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999437382 0 0) 0x7f975c1d7710 con 0x7f975c13f710 2024-03-25T19:48:38.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.018+0000 7f974affd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:38.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.018+0000 7f974affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:38.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.018+0000 7f974affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:38.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.018+0000 7f974affd640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:48:38.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.018+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c004eb0 con 0x7f975c13f710 2024-03-25T19:48:38.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3080561025 0 0) 0x7f975c1d8340 con 0x7f975c1430c0 2024-03-25T19:48:38.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 >> v1:172.21.15.18:6790/0 conn(0x7f975c0a98f0 legacy=0x7f975c137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 >> v1:172.21.15.53:6789/0 conn(0x7f975c13f710 legacy=0x7f975c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:38.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 1 monclient: found mon.a 2024-03-25T19:48:38.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:38.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c1d84d0 con 0x7f975c1430c0 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 10 monclient: _finish_auth 0 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f974affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 5 monclient: authenticate success, global_id 4575 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 10 monclient: _renew_subs 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975c1d41e0 con 0x7f975c1430c0 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 10 monclient: _renew_subs 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f975c1d5150 con 0x7f975c1430c0 2024-03-25T19:48:38.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.020+0000 7f9760bc4640 1 librados: init done 2024-03-25T19:48:38.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.021+0000 7f9760bc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:38.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.021+0000 7f9760bc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:38.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.021+0000 7f9760bc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:38.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.021+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975c0627b0 con 0x7f975c1430c0 2024-03-25T19:48:38.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.247+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9754003990 con 0x7f975c1430c0 2024-03-25T19:48:38.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.247+0000 7f974affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:38.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.247+0000 7f9758d99640 4 set_mon_vals no callback set 2024-03-25T19:48:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.341+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9754003da0 con 0x7f975c1430c0 2024-03-25T19:48:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.341+0000 7f974affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:38.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.341+0000 7f974affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.341+0000 7f974affd640 10 monclient: dump: 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:38.342 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:38.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.342+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f9754003f80 con 0x7f975c1430c0 2024-03-25T19:48:38.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.343+0000 7f975a59c640 10 monclient: get_auth_request con 0x7f973c06de20 auth_method 0 2024-03-25T19:48:38.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.343+0000 7f974affd640 10 client.4575.objecter ms_handle_connect 0x7f973c06de20 2024-03-25T19:48:38.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.360+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f9754054af0 con 0x7f975c1430c0 2024-03-25T19:48:38.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.360+0000 7f974affd640 10 client.4575.objecter ms_dispatch 0x7f975c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:38.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.360+0000 7f974affd640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:38.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.360+0000 7f974affd640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:38.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.360+0000 7f974affd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T19:48:38.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.360+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9754050600 con 0x7f975c1430c0 2024-03-25T19:48:38.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.360+0000 7f974affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:38.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.360+0000 7f974affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:38.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.620+0000 7f9760bc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.620+0000 7f9760bc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.620+0000 7f9760bc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:38.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.620+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f975c069f40 con 0x7f975c1430c0 2024-03-25T19:48:38.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.652+0000 7f974affd640 1 -- 172.21.15.18:0/2793778444 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f9754052530 con 0x7f975c1430c0 2024-03-25T19:48:38.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.652+0000 7f974affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.652+0000 7f974affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:38.653 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:48:38.653 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.654+0000 7f9760bc4640 10 librados: watch_flush enter 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.654+0000 7f9760bc4640 10 librados: watch_flush exit 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.654+0000 7f9760bc4640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.654+0000 7f9760bc4640 10 client.4575.objecter successfully canceled tick 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 >> v1:172.21.15.18:6800/1520000573 conn(0x7f973c06de20 legacy=0x7f973c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 10 monclient: shutdown 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 >> v1:172.21.15.18:6789/0 conn(0x7f975c1430c0 legacy=0x7f975c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 shutdown_connections 2024-03-25T19:48:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 >> 172.21.15.18:0/2793778444 conn(0x7f975c0b7ea0 msgr2=0x7f975c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 shutdown_connections 2024-03-25T19:48:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 1 -- 172.21.15.18:0/2793778444 wait complete. 2024-03-25T19:48:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:38.655+0000 7f9760bc4640 1 librados: shutdown 2024-03-25T19:48:38.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:45.475 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:45.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.591+0000 7f7f6cb96640 10 monclient: get_monmap_and_config 2024-03-25T19:48:45.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.591+0000 7f7f6cb96640 10 monclient: build_initial_monmap 2024-03-25T19:48:45.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.591+0000 7f7f6cb96640 10 monclient: monmap: 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:48:45.592661+0000 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:48:45.592661+0000 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:48:45.592 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:45.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.592+0000 7f7f6cb96640 1 Processor -- start 2024-03-25T19:48:45.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.592+0000 7f7f6cb96640 1 -- start start 2024-03-25T19:48:45.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.592+0000 7f7f6cb96640 10 monclient: init 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 10 monclient(hunting): picked mon.noname-a con 0x7f7f680b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 10 monclient(hunting): picked mon.noname-c con 0x7f7f680b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 10 monclient(hunting): picked mon.noname-b con 0x7f7f6814be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f680af940 con 0x7f7f680b4d60 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f680b0a70 con 0x7f7f680b1730 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f680b0d50 con 0x7f7f6814be90 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.593+0000 7f7f6cb96640 10 monclient(hunting): authenticate will time out at 5352.720729s 2024-03-25T19:48:45.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.602+0000 7f7f66d9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f7f6814be90 0x7f7f6814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:48148/0 (socket says 172.21.15.18:48148) 2024-03-25T19:48:45.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.602+0000 7f7f66d9d640 1 -- 172.21.15.18:0/1261841935 learned_addr learned my addr 172.21.15.18:0/1261841935 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:45.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.610+0000 7f7f65d9b640 1 -- 172.21.15.18:0/1261841935 >> v1:172.21.15.53:6789/0 conn(0x7f7f680b1730 legacy=0x7f7f680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T19:48:45.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.610+0000 7f7f65d9b640 1 -- 172.21.15.18:0/1261841935 >> v1:172.21.15.53:6789/0 conn(0x7f7f680b1730 legacy=0x7f7f680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:48:45.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.610+0000 7f7f65d9b640 1 --1- 172.21.15.18:0/1261841935 >> v1:172.21.15.53:6789/0 conn(0x7f7f680b1730 0x7f7f680b1be0 :-1 s=OPENED pgs=55818 cs=1 l=1).handle_message read tag failed 2024-03-25T19:48:45.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.610+0000 7f7f65d9b640 1 --1- 172.21.15.18:0/1261841935 >> v1:172.21.15.53:6789/0 conn(0x7f7f680b1730 0x7f7f680b1be0 :-1 s=OPENED pgs=55818 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:48:45.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.610+0000 7f7f6559a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.53:6789/0 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.615+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725631890 0 0) 0x7f7f680b0d50 con 0x7f7f6814be90 2024-03-25T19:48:45.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.615+0000 7f7f6559a640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.615+0000 7f7f6559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.615+0000 7f7f6559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.615+0000 7f7f6559a640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:48:45.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.615+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f50003520 con 0x7f7f6814be90 2024-03-25T19:48:45.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.624+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282241910 0 0) 0x7f7f680af940 con 0x7f7f680b4d60 2024-03-25T19:48:45.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.624+0000 7f7f6559a640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.624+0000 7f7f6559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.624+0000 7f7f6559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.624+0000 7f7f6559a640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:48:45.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.624+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f680b0d50 con 0x7f7f680b4d60 2024-03-25T19:48:45.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3666975558 0 0) 0x7f7f50003520 con 0x7f7f6814be90 2024-03-25T19:48:45.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 >> v1:172.21.15.53:6789/0 conn(0x7f7f680b1730 legacy=0x7f7f680b1be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:48:45.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 >> v1:172.21.15.18:6789/0 conn(0x7f7f680b4d60 legacy=0x7f7f6813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 10 monclient: _finish_hunting 0 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 1 monclient: found mon.noname-b 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f680b1030 con 0x7f7f6814be90 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 10 monclient: _finish_auth 0 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 10 monclient: _check_auth_tickets 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7f60001a10 con 0x7f7f6814be90 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6559a640 10 monclient: dump: 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:45.629 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.630 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:45.630 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:45.630 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:45.630 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:45.630 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:45.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6cb96640 5 monclient: authenticate success, global_id 4547 2024-03-25T19:48:45.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.628+0000 7f7f6cb96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.634+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f60001b70 con 0x7f7f6814be90 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.634+0000 7f7f6559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.634+0000 7f7f6559a640 1 -- 172.21.15.18:0/1261841935 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7f60001f80 con 0x7f7f6814be90 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.634+0000 7f7f6cb96640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.634+0000 7f7f6cb96640 4 set_mon_vals no callback set 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.634+0000 7f7f6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.634+0000 7f7f6559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.634+0000 7f7f6559a640 10 monclient: dump: 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:45.634 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.635 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:45.635 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:45.635 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:45.635 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:45.635 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.635+0000 7f7f6cb96640 10 monclient: shutdown 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.635+0000 7f7f6cb96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.635+0000 7f7f6cb96640 1 -- 172.21.15.18:0/1261841935 >> v1:172.21.15.18:6790/0 conn(0x7f7f6814be90 legacy=0x7f7f6814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.635+0000 7f7f6cb96640 1 -- 172.21.15.18:0/1261841935 shutdown_connections 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.635+0000 7f7f6cb96640 1 -- 172.21.15.18:0/1261841935 >> 172.21.15.18:0/1261841935 conn(0x7f7f680617a0 msgr2=0x7f7f68061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.635+0000 7f7f6cb96640 1 -- 172.21.15.18:0/1261841935 shutdown_connections 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.635+0000 7f7f6cb96640 1 -- 172.21.15.18:0/1261841935 wait complete. 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 10 monclient: build_initial_monmap 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 10 monclient: monmap: 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:45.636 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 1 librados: starting msgr at 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 1 librados: starting objecter 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 1 Processor -- start 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 1 -- start start 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 1 librados: setting wanted keys 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 1 librados: calling monclient init 2024-03-25T19:48:45.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.636+0000 7f7f6cb96640 10 monclient: init 2024-03-25T19:48:45.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:45.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:45.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 10 monclient(hunting): picked mon.noname-c con 0x7f7f680b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:45.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 10 monclient(hunting): picked mon.noname-a con 0x7f7f680b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:45.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 10 monclient(hunting): picked mon.noname-b con 0x7f7f6814be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:45.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f681d78d0 con 0x7f7f680b4d60 2024-03-25T19:48:45.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f681d8500 con 0x7f7f6814be90 2024-03-25T19:48:45.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f68130570 con 0x7f7f680b1730 2024-03-25T19:48:45.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:45.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.637+0000 7f7f6cb96640 10 monclient(hunting): authenticate will time out at 5352.764171s 2024-03-25T19:48:45.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.649+0000 7f7f66d9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7f6814be90 0x7f7f68136db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:50936/0 (socket says 172.21.15.18:50936) 2024-03-25T19:48:45.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.649+0000 7f7f66d9d640 1 -- 172.21.15.18:0/2837356852 learned_addr learned my addr 172.21.15.18:0/2837356852 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:45.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.651+0000 7f7f56ffd640 10 client.?.objecter ms_handle_connect 0x7f7f6814be90 2024-03-25T19:48:45.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.651+0000 7f7f56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.654+0000 7f7f56ffd640 10 client.?.objecter ms_handle_connect 0x7f7f680b4d60 2024-03-25T19:48:45.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.654+0000 7f7f56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.657+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062733016 0 0) 0x7f7f681d78d0 con 0x7f7f680b4d60 2024-03-25T19:48:45.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.657+0000 7f7f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.657+0000 7f7f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.657+0000 7f7f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.657+0000 7f7f56ffd640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:48:45.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.657+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f48004eb0 con 0x7f7f680b4d60 2024-03-25T19:48:45.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.659+0000 7f7f56ffd640 10 client.?.objecter ms_handle_connect 0x7f7f680b1730 2024-03-25T19:48:45.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.659+0000 7f7f56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.667+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384766856 0 0) 0x7f7f681d8500 con 0x7f7f6814be90 2024-03-25T19:48:45.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.667+0000 7f7f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.667+0000 7f7f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.667+0000 7f7f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.667+0000 7f7f56ffd640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:48:45.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.667+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f681d78d0 con 0x7f7f6814be90 2024-03-25T19:48:45.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.670+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3210047228 0 0) 0x7f7f48004eb0 con 0x7f7f680b4d60 2024-03-25T19:48:45.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.670+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f681d8500 con 0x7f7f680b4d60 2024-03-25T19:48:45.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.670+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7f5c001960 con 0x7f7f680b4d60 2024-03-25T19:48:45.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.670+0000 7f7f56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.670+0000 7f7f56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.670+0000 7f7f56ffd640 10 monclient(hunting): dump: 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:45.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.670+0000 7f7f56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:45.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.684+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489826662 0 0) 0x7f7f68130570 con 0x7f7f680b1730 2024-03-25T19:48:45.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.685+0000 7f7f56ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.685+0000 7f7f56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.685+0000 7f7f56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.685+0000 7f7f56ffd640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:48:45.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.685+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f48004eb0 con 0x7f7f680b1730 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 14856541 0 0) 0x7f7f681d8500 con 0x7f7f680b4d60 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 >> v1:172.21.15.18:6790/0 conn(0x7f7f680b1730 legacy=0x7f7f681d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 >> v1:172.21.15.53:6789/0 conn(0x7f7f6814be90 legacy=0x7f7f68136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 1 monclient: found mon.a 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f68130700 con 0x7f7f680b4d60 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 10 monclient: _finish_auth 0 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 5 monclient: authenticate success, global_id 4581 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 10 monclient: _renew_subs 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:45.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f681374c0 con 0x7f7f680b4d60 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 10 monclient: _renew_subs 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f68136840 con 0x7f7f680b4d60 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.696+0000 7f7f6cb96640 1 librados: init done 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.697+0000 7f7f6cb96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.697+0000 7f7f6cb96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.697+0000 7f7f6cb96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:45.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:45.697+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f681401f0 con 0x7f7f680b4d60 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.012+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f5c0035f0 con 0x7f7f680b4d60 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.012+0000 7f7f56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.012+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7f5c003980 con 0x7f7f680b4d60 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.012+0000 7f7f56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.012+0000 7f7f56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.012+0000 7f7f56ffd640 10 monclient: dump: 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:46.012 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:46.013 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:46.013 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:46.013 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:46.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.012+0000 7f7f64d99640 4 set_mon_vals no callback set 2024-03-25T19:48:46.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.012+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f7f5c003ba0 con 0x7f7f680b4d60 2024-03-25T19:48:46.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.013+0000 7f7f6659c640 10 monclient: get_auth_request con 0x7f7f4806de20 auth_method 0 2024-03-25T19:48:46.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.013+0000 7f7f56ffd640 10 client.4581.objecter ms_handle_connect 0x7f7f4806de20 2024-03-25T19:48:46.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.021+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f7f5c054810 con 0x7f7f680b4d60 2024-03-25T19:48:46.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.021+0000 7f7f56ffd640 10 client.4581.objecter ms_dispatch 0x7f7f68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:46.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.021+0000 7f7f56ffd640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:46.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.021+0000 7f7f56ffd640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.021+0000 7f7f56ffd640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T19:48:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.021+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f5c050320 con 0x7f7f680b4d60 2024-03-25T19:48:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.021+0000 7f7f56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:46.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.021+0000 7f7f56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:46.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.228+0000 7f7f6cb96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.228+0000 7f7f6cb96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.228+0000 7f7f6cb96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:48:46.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.228+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f680627b0 con 0x7f7f680b4d60 2024-03-25T19:48:46.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.281+0000 7f7f56ffd640 1 -- 172.21.15.18:0/2837356852 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f7f5c052250 con 0x7f7f680b4d60 2024-03-25T19:48:46.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.282+0000 7f7f56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.282+0000 7f7f56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:46.282 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:48:46.282 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:46.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 10 librados: watch_flush enter 2024-03-25T19:48:46.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 10 librados: watch_flush exit 2024-03-25T19:48:46.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T19:48:46.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 10 client.4581.objecter successfully canceled tick 2024-03-25T19:48:46.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 >> v1:172.21.15.18:6800/1520000573 conn(0x7f7f4806de20 legacy=0x7f7f480702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:46.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 10 monclient: shutdown 2024-03-25T19:48:46.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:46.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 >> v1:172.21.15.18:6789/0 conn(0x7f7f680b4d60 legacy=0x7f7f6812fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:46.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 shutdown_connections 2024-03-25T19:48:46.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 >> 172.21.15.18:0/2837356852 conn(0x7f7f680617a0 msgr2=0x7f7f6813c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:46.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 shutdown_connections 2024-03-25T19:48:46.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 1 -- 172.21.15.18:0/2837356852 wait complete. 2024-03-25T19:48:46.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:46.284+0000 7f7f6cb96640 1 librados: shutdown 2024-03-25T19:48:46.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:53.100 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:53.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.220+0000 7f12c5458640 10 monclient: get_monmap_and_config 2024-03-25T19:48:53.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.220+0000 7f12c5458640 10 monclient: build_initial_monmap 2024-03-25T19:48:53.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.220+0000 7f12c5458640 10 monclient: monmap: 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:48:53.221597+0000 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:48:53.221597+0000 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:48:53.221 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:53.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.221+0000 7f12c5458640 1 Processor -- start 2024-03-25T19:48:53.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.221+0000 7f12c5458640 1 -- start start 2024-03-25T19:48:53.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.221+0000 7f12c5458640 10 monclient: init 2024-03-25T19:48:53.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 10 monclient(hunting): picked mon.noname-a con 0x7f12c00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 10 monclient(hunting): picked mon.noname-c con 0x7f12c013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 10 monclient(hunting): picked mon.noname-b con 0x7f12c01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c0142520 con 0x7f12c00a98f0 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c00b28a0 con 0x7f12c013f710 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c00b2a30 con 0x7f12c01430c0 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.222+0000 7f12c5458640 10 monclient(hunting): authenticate will time out at 5360.349080s 2024-03-25T19:48:53.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.236+0000 7f12beffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f12c00a98f0 0x7f12c00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:51992/0 (socket says 172.21.15.18:51992) 2024-03-25T19:48:53.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.236+0000 7f12beffd640 1 -- 172.21.15.18:0/3276763374 learned_addr learned my addr 172.21.15.18:0/3276763374 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:53.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.248+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340260974 0 0) 0x7f12c00b28a0 con 0x7f12c013f710 2024-03-25T19:48:53.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.248+0000 7f12be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.248+0000 7f12be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.248+0000 7f12be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.248+0000 7f12be7fc640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:48:53.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.248+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ac003520 con 0x7f12c013f710 2024-03-25T19:48:53.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.252+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736878284 0 0) 0x7f12c0142520 con 0x7f12c00a98f0 2024-03-25T19:48:53.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.252+0000 7f12be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.252+0000 7f12be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.252+0000 7f12be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.252+0000 7f12be7fc640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:48:53.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.252+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c00b28a0 con 0x7f12c00a98f0 2024-03-25T19:48:53.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.256+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166658771 0 0) 0x7f12c00b2a30 con 0x7f12c01430c0 2024-03-25T19:48:53.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:48:53.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c0142520 con 0x7f12c01430c0 2024-03-25T19:48:53.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3533943275 0 0) 0x7f12ac003520 con 0x7f12c013f710 2024-03-25T19:48:53.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 >> v1:172.21.15.18:6790/0 conn(0x7f12c01430c0 legacy=0x7f12c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 >> v1:172.21.15.18:6789/0 conn(0x7f12c00a98f0 legacy=0x7f12c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:53.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 1 monclient: found mon.noname-c 2024-03-25T19:48:53.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c00b44c0 con 0x7f12c013f710 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f12a8001690 con 0x7f12c013f710 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12be7fc640 10 monclient: dump: 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.259 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.260 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:53.260 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:53.260 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:53.260 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:53.260 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:53.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12c5458640 5 monclient: authenticate success, global_id 4552 2024-03-25T19:48:53.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.257+0000 7f12c5458640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:53.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.276+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12a80017f0 con 0x7f12c013f710 2024-03-25T19:48:53.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.276+0000 7f12be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:53.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.276+0000 7f12be7fc640 1 -- 172.21.15.18:0/3276763374 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f12a8001c00 con 0x7f12c013f710 2024-03-25T19:48:53.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.276+0000 7f12c5458640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:53.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.276+0000 7f12c5458640 4 set_mon_vals no callback set 2024-03-25T19:48:53.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.276+0000 7f12be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.276+0000 7f12be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.276+0000 7f12be7fc640 10 monclient: dump: 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:53.277 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.277+0000 7f12c5458640 10 monclient: shutdown 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.277+0000 7f12c5458640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.277+0000 7f12c5458640 1 -- 172.21.15.18:0/3276763374 >> v1:172.21.15.53:6789/0 conn(0x7f12c013f710 legacy=0x7f12c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.277+0000 7f12c5458640 1 -- 172.21.15.18:0/3276763374 shutdown_connections 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.277+0000 7f12c5458640 1 -- 172.21.15.18:0/3276763374 >> 172.21.15.18:0/3276763374 conn(0x7f12c00b7ea0 msgr2=0x7f12c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.277+0000 7f12c5458640 1 -- 172.21.15.18:0/3276763374 shutdown_connections 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.277+0000 7f12c5458640 1 -- 172.21.15.18:0/3276763374 wait complete. 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.278+0000 7f12c5458640 10 monclient: build_initial_monmap 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.278+0000 7f12c5458640 10 monclient: monmap: 2024-03-25T19:48:53.279 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.279+0000 7f12c5458640 1 librados: starting msgr at 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.279+0000 7f12c5458640 1 librados: starting objecter 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.279+0000 7f12c5458640 1 Processor -- start 2024-03-25T19:48:53.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.279+0000 7f12c5458640 1 -- start start 2024-03-25T19:48:53.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.279+0000 7f12c5458640 1 librados: setting wanted keys 2024-03-25T19:48:53.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.279+0000 7f12c5458640 1 librados: calling monclient init 2024-03-25T19:48:53.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.279+0000 7f12c5458640 10 monclient: init 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 10 monclient(hunting): picked mon.noname-b con 0x7f12c00a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 10 monclient(hunting): picked mon.noname-c con 0x7f12c013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 10 monclient(hunting): picked mon.noname-a con 0x7f12c01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c0139070 con 0x7f12c01430c0 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c0133710 con 0x7f12c00a98f0 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c01338a0 con 0x7f12c013f710 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:53.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.280+0000 7f12c5458640 10 monclient(hunting): authenticate will time out at 5360.407164s 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.286+0000 7f12bf7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f12c01430c0 0x7f12c0133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:52040/0 (socket says 172.21.15.18:52040) 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.286+0000 7f12bf7fe640 1 -- 172.21.15.18:0/2485112640 learned_addr learned my addr 172.21.15.18:0/2485112640 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:48:53.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.290+0000 7f12b77fe640 10 client.?.objecter ms_handle_connect 0x7f12c013f710 2024-03-25T19:48:53.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.290+0000 7f12b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.291+0000 7f12b77fe640 10 client.?.objecter ms_handle_connect 0x7f12c01430c0 2024-03-25T19:48:53.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.291+0000 7f12b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.292+0000 7f12b77fe640 10 client.?.objecter ms_handle_connect 0x7f12c00a98f0 2024-03-25T19:48:53.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.292+0000 7f12b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.305+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871750265 0 0) 0x7f12c01338a0 con 0x7f12c013f710 2024-03-25T19:48:53.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.305+0000 7f12b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.305+0000 7f12b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.305+0000 7f12b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.305+0000 7f12b77fe640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:48:53.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.305+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a0004eb0 con 0x7f12c013f710 2024-03-25T19:48:53.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.307+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682647279 0 0) 0x7f12c0139070 con 0x7f12c01430c0 2024-03-25T19:48:53.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:48:53.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c01338a0 con 0x7f12c01430c0 2024-03-25T19:48:53.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938797116 0 0) 0x7f12c0133710 con 0x7f12c00a98f0 2024-03-25T19:48:53.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:48:53.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.308+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c0139070 con 0x7f12c00a98f0 2024-03-25T19:48:53.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.314+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 161139400 0 0) 0x7f12a0004eb0 con 0x7f12c013f710 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.314+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12c0133710 con 0x7f12c013f710 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.314+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f12a8001b00 con 0x7f12c013f710 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.314+0000 7f12b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.314+0000 7f12b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.314+0000 7f12b77fe640 10 monclient(hunting): dump: 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:53.314 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:53.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.314+0000 7f12b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.330+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2182756358 0 0) 0x7f12c0139070 con 0x7f12c00a98f0 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.330+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12a0004eb0 con 0x7f12c00a98f0 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.330+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f12b0001930 con 0x7f12c00a98f0 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.330+0000 7f12b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.330+0000 7f12b77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.330+0000 7f12b77fe640 10 monclient(hunting): dump: 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.330 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.331 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:53.331 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:53.331 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:53.331 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:53.331 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.335+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2765278626 0 0) 0x7f12c01338a0 con 0x7f12c01430c0 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.335+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12c0139070 con 0x7f12c01430c0 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.335+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f12b8001c50 con 0x7f12c01430c0 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.335+0000 7f12b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.335+0000 7f12b77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.335+0000 7f12b77fe640 10 monclient(hunting): dump: 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.335 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:53.336 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:53.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 710061221 0 0) 0x7f12c0133710 con 0x7f12c013f710 2024-03-25T19:48:53.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 >> v1:172.21.15.53:6789/0 conn(0x7f12c00a98f0 legacy=0x7f12c01378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 >> v1:172.21.15.18:6789/0 conn(0x7f12c01430c0 legacy=0x7f12c0133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:53.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 1 monclient: found mon.c 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c0133a30 con 0x7f12c013f710 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 5 monclient: authenticate success, global_id 4556 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 10 monclient: _renew_subs 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c00b7100 con 0x7f12c013f710 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 10 monclient: _renew_subs 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:53.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c0136e60 con 0x7f12c013f710 2024-03-25T19:48:53.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.338+0000 7f12c5458640 1 librados: init done 2024-03-25T19:48:53.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.339+0000 7f12c5458640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.339+0000 7f12c5458640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.339+0000 7f12c5458640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:53.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.339+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c0074cc0 con 0x7f12c013f710 2024-03-25T19:48:53.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.678+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12a80038a0 con 0x7f12c013f710 2024-03-25T19:48:53.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.678+0000 7f12b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.678+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f12a8003bb0 con 0x7f12c013f710 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.678+0000 7f12b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.678+0000 7f12b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.678+0000 7f12b77fe640 10 monclient: dump: 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:48:53.679 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:48:53.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.678+0000 7f12bdffb640 4 set_mon_vals no callback set 2024-03-25T19:48:53.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.679+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f12a8019b80 con 0x7f12c013f710 2024-03-25T19:48:53.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.680+0000 7f12beffd640 10 monclient: get_auth_request con 0x7f12a006e0f0 auth_method 0 2024-03-25T19:48:53.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.680+0000 7f12b77fe640 10 client.4556.objecter ms_handle_connect 0x7f12a006e0f0 2024-03-25T19:48:53.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.686+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f12a8053650 con 0x7f12c013f710 2024-03-25T19:48:53.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.686+0000 7f12b77fe640 10 client.4556.objecter ms_dispatch 0x7f12c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:53.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.686+0000 7f12b77fe640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:53.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.686+0000 7f12b77fe640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:53.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.686+0000 7f12b77fe640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T19:48:53.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.686+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12a804ef90 con 0x7f12c013f710 2024-03-25T19:48:53.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.686+0000 7f12b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.686+0000 7f12b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:53.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.899+0000 7f12c5458640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.899+0000 7f12c5458640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.899+0000 7f12c5458640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:48:53.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.899+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12c00627b0 con 0x7f12c013f710 2024-03-25T19:48:53.907 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:21.437 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:53.917 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:21.447 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:53.963 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:21.494 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:53.966 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:21.462 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:53.982 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:21.478 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:53.991 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:21.522 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:53.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.992+0000 7f12b77fe640 1 -- 172.21.15.18:0/2485112640 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f12a8051090 con 0x7f12c013f710 2024-03-25T19:48:53.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.992+0000 7f12b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.992+0000 7f12b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:53.993 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:48:53.993 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:53.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 10 librados: watch_flush enter 2024-03-25T19:48:53.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 10 librados: watch_flush exit 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 10 client.4556.objecter successfully canceled tick 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 >> v1:172.21.15.18:6800/1520000573 conn(0x7f12a006e0f0 legacy=0x7f12a00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 10 monclient: shutdown 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 >> v1:172.21.15.18:6790/0 conn(0x7f12c013f710 legacy=0x7f12c01daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 shutdown_connections 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 >> 172.21.15.18:0/2485112640 conn(0x7f12c00b7ea0 msgr2=0x7f12c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 shutdown_connections 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 1 -- 172.21.15.18:0/2485112640 wait complete. 2024-03-25T19:48:53.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:48:53.995+0000 7f12c5458640 1 librados: shutdown 2024-03-25T19:48:54.002 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:21.532 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:54.002 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:21.498 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:54.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:54.035 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:21.565 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.813 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:00.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.925+0000 7f791256b640 10 monclient: get_monmap_and_config 2024-03-25T19:49:00.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.925+0000 7f791256b640 10 monclient: build_initial_monmap 2024-03-25T19:49:00.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.925+0000 7f791256b640 10 monclient: monmap: 2024-03-25T19:49:00.925 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:00.925 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:00.925 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:49:00.926474+0000 2024-03-25T19:49:00.926 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:49:00.926474+0000 2024-03-25T19:49:00.926 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:00.926 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:00.926 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:00.926 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:49:00.926 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:49:00.926 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.926+0000 7f791256b640 1 Processor -- start 2024-03-25T19:49:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.926+0000 7f791256b640 1 -- start start 2024-03-25T19:49:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.926+0000 7f791256b640 10 monclient: init 2024-03-25T19:49:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.926+0000 7f791256b640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.926+0000 7f791256b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.926+0000 7f791256b640 10 monclient(hunting): picked mon.noname-c con 0x7f790c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.926+0000 7f791256b640 10 monclient(hunting): picked mon.noname-b con 0x7f790c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.927+0000 7f791256b640 10 monclient(hunting): picked mon.noname-a con 0x7f790c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.927+0000 7f791256b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c142520 con 0x7f790c1430c0 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.927+0000 7f791256b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c0b28a0 con 0x7f790c0a98f0 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.927+0000 7f791256b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c0b2a30 con 0x7f790c13f710 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.927+0000 7f791256b640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:00.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.927+0000 7f791256b640 10 monclient(hunting): authenticate will time out at 5368.053806s 2024-03-25T19:49:00.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.958+0000 7f790bfff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f790c0a98f0 0x7f790c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:56652/0 (socket says 172.21.15.18:56652) 2024-03-25T19:49:00.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.958+0000 7f790bfff640 1 -- 172.21.15.18:0/4023027445 learned_addr learned my addr 172.21.15.18:0/4023027445 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:00.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.971+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654584808 0 0) 0x7f790c0b28a0 con 0x7f790c0a98f0 2024-03-25T19:49:00.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.971+0000 7f790affd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:00.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.971+0000 7f790affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:00.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.971+0000 7f790affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:00.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.971+0000 7f790affd640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:49:00.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.971+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f4003520 con 0x7f790c0a98f0 2024-03-25T19:49:00.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.979+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297479867 0 0) 0x7f790c0b2a30 con 0x7f790c13f710 2024-03-25T19:49:00.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.979+0000 7f790affd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:00.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.979+0000 7f790affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:00.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.979+0000 7f790affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:00.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.979+0000 7f790affd640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:49:00.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.979+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f790c0b28a0 con 0x7f790c13f710 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.990+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714795265 0 0) 0x7f78f4003520 con 0x7f790c0a98f0 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.990+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 >> v1:172.21.15.18:6790/0 conn(0x7f790c13f710 legacy=0x7f790c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.990+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 >> v1:172.21.15.18:6789/0 conn(0x7f790c1430c0 legacy=0x7f790c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.990+0000 7f790affd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.990+0000 7f790affd640 1 monclient: found mon.noname-c 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c0b44c0 con 0x7f790c0a98f0 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 10 monclient: _finish_auth 0 2024-03-25T19:49:00.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f78fc001690 con 0x7f790c0a98f0 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f790affd640 10 monclient: dump: 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:00.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f791256b640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:49:00.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:00.991+0000 7f791256b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.004+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78fc0017f0 con 0x7f790c0a98f0 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.004+0000 7f790affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.004+0000 7f790affd640 1 -- 172.21.15.18:0/4023027445 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f78fc001c00 con 0x7f790c0a98f0 2024-03-25T19:49:01.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.004+0000 7f791256b640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.004+0000 7f791256b640 4 set_mon_vals no callback set 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.004+0000 7f790affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.004+0000 7f790affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.004+0000 7f790affd640 10 monclient: dump: 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:01.005 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.005+0000 7f791256b640 10 monclient: shutdown 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.005+0000 7f791256b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.005+0000 7f791256b640 1 -- 172.21.15.18:0/4023027445 >> v1:172.21.15.53:6789/0 conn(0x7f790c0a98f0 legacy=0x7f790c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.005+0000 7f791256b640 1 -- 172.21.15.18:0/4023027445 shutdown_connections 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.005+0000 7f791256b640 1 -- 172.21.15.18:0/4023027445 >> 172.21.15.18:0/4023027445 conn(0x7f790c0b7ea0 msgr2=0x7f790c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.005+0000 7f791256b640 1 -- 172.21.15.18:0/4023027445 shutdown_connections 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.005+0000 7f791256b640 1 -- 172.21.15.18:0/4023027445 wait complete. 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.006+0000 7f791256b640 10 monclient: build_initial_monmap 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.006+0000 7f791256b640 10 monclient: monmap: 2024-03-25T19:49:01.007 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.006+0000 7f791256b640 1 librados: starting msgr at 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.006+0000 7f791256b640 1 librados: starting objecter 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.006+0000 7f791256b640 1 Processor -- start 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.007+0000 7f791256b640 1 -- start start 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.007+0000 7f791256b640 1 librados: setting wanted keys 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.007+0000 7f791256b640 1 librados: calling monclient init 2024-03-25T19:49:01.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.007+0000 7f791256b640 10 monclient: init 2024-03-25T19:49:01.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 10 monclient(hunting): picked mon.noname-b con 0x7f790c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 10 monclient(hunting): picked mon.noname-a con 0x7f790c13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 10 monclient(hunting): picked mon.noname-c con 0x7f790c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c139a40 con 0x7f790c13f710 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c133f20 con 0x7f790c0a98f0 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c0ab950 con 0x7f790c1430c0 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.008+0000 7f791256b640 10 monclient(hunting): authenticate will time out at 5368.134958s 2024-03-25T19:49:01.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.011+0000 7f7910b08640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f790c1430c0 0x7f790c133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:59220/0 (socket says 172.21.15.18:59220) 2024-03-25T19:49:01.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.011+0000 7f7910b08640 1 -- 172.21.15.18:0/2691577294 learned_addr learned my addr 172.21.15.18:0/2691577294 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:01.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.015+0000 7f7908ff9640 10 client.?.objecter ms_handle_connect 0x7f790c1430c0 2024-03-25T19:49:01.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.015+0000 7f7908ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:01.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.022+0000 7f7908ff9640 10 client.?.objecter ms_handle_connect 0x7f790c13f710 2024-03-25T19:49:01.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.022+0000 7f7908ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:01.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.025+0000 7f7908ff9640 10 client.?.objecter ms_handle_connect 0x7f790c0a98f0 2024-03-25T19:49:01.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.025+0000 7f7908ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:01.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.028+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149702626 0 0) 0x7f790c139a40 con 0x7f790c13f710 2024-03-25T19:49:01.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.028+0000 7f7908ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.028+0000 7f7908ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.028+0000 7f7908ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.028+0000 7f7908ff9640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:49:01.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.028+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f8004eb0 con 0x7f790c13f710 2024-03-25T19:49:01.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.038+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183111265 0 0) 0x7f790c133f20 con 0x7f790c0a98f0 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.038+0000 7f7908ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.038+0000 7f7908ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.038+0000 7f7908ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.038+0000 7f7908ff9640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.038+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f790c139a40 con 0x7f790c0a98f0 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.039+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299036107 0 0) 0x7f790c0ab950 con 0x7f790c1430c0 2024-03-25T19:49:01.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.039+0000 7f7908ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.039+0000 7f7908ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.039+0000 7f7908ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.039+0000 7f7908ff9640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:49:01.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.039+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f790c133f20 con 0x7f790c1430c0 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.049+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769537531 0 0) 0x7f78f8004eb0 con 0x7f790c13f710 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.049+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f790c0ab950 con 0x7f790c13f710 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.049+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7900001930 con 0x7f790c13f710 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.049+0000 7f7908ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.049+0000 7f7908ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.049+0000 7f7908ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:01.049 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:01.050 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:01.050 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:01.050 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:01.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.049+0000 7f7908ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:01.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1557303227 0 0) 0x7f790c0ab950 con 0x7f790c13f710 2024-03-25T19:49:01.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 >> v1:172.21.15.18:6790/0 conn(0x7f790c1430c0 legacy=0x7f790c133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 >> v1:172.21.15.53:6789/0 conn(0x7f790c0a98f0 legacy=0x7f790c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:01.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 1 monclient: found mon.a 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c0abc30 con 0x7f790c13f710 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f7908ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 5 monclient: authenticate success, global_id 4593 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 10 monclient: _renew_subs 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c0abf10 con 0x7f790c13f710 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 10 monclient: _renew_subs 2024-03-25T19:49:01.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:49:01.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c139370 con 0x7f790c13f710 2024-03-25T19:49:01.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.057+0000 7f791256b640 1 librados: init done 2024-03-25T19:49:01.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.058+0000 7f791256b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.058+0000 7f791256b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.058+0000 7f791256b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:49:01.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.058+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c0627b0 con 0x7f790c13f710 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.369+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7900003610 con 0x7f790c13f710 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.369+0000 7f7908ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.369+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f7900003a20 con 0x7f790c13f710 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.369+0000 7f7908ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.369+0000 7f7908ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.369+0000 7f7908ff9640 10 monclient: dump: 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:01.369 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:01.370 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.370 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:01.370 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:01.370 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:01.370 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:01.370 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:01.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.369+0000 7f790a7fc640 4 set_mon_vals no callback set 2024-03-25T19:49:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.370+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f7900003c40 con 0x7f790c13f710 2024-03-25T19:49:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.370+0000 7f790bfff640 10 monclient: get_auth_request con 0x7f78f806ddf0 auth_method 0 2024-03-25T19:49:01.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.370+0000 7f7908ff9640 10 client.4593.objecter ms_handle_connect 0x7f78f806ddf0 2024-03-25T19:49:01.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.376+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f7900053660 con 0x7f790c13f710 2024-03-25T19:49:01.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.376+0000 7f7908ff9640 10 client.4593.objecter ms_dispatch 0x7f790c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:01.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.376+0000 7f7908ff9640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:01.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.376+0000 7f7908ff9640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:01.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.376+0000 7f7908ff9640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T19:49:01.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.376+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f790004f170 con 0x7f790c13f710 2024-03-25T19:49:01.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.376+0000 7f7908ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.376+0000 7f7908ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:01.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.639+0000 7f791256b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.639+0000 7f791256b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.639+0000 7f791256b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:49:01.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.639+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f790c0b7180 con 0x7f790c13f710 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.667+0000 7f7908ff9640 1 -- 172.21.15.18:0/2691577294 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f79000510a0 con 0x7f790c13f710 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.667+0000 7f7908ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.667+0000 7f7908ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:01.667 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:49:01.668 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 10 librados: watch_flush enter 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 10 librados: watch_flush exit 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 10 client.4593.objecter successfully canceled tick 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 >> v1:172.21.15.18:6800/1520000573 conn(0x7f78f806ddf0 legacy=0x7f78f80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 10 monclient: shutdown 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 >> v1:172.21.15.18:6789/0 conn(0x7f790c13f710 legacy=0x7f790c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 shutdown_connections 2024-03-25T19:49:01.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 >> 172.21.15.18:0/2691577294 conn(0x7f790c0b7ea0 msgr2=0x7f790c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:01.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 shutdown_connections 2024-03-25T19:49:01.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 1 -- 172.21.15.18:0/2691577294 wait complete. 2024-03-25T19:49:01.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:01.670+0000 7f791256b640 1 librados: shutdown 2024-03-25T19:49:01.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:08.487 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:08.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.603+0000 7fea79e81640 10 monclient: get_monmap_and_config 2024-03-25T19:49:08.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.603+0000 7fea79e81640 10 monclient: build_initial_monmap 2024-03-25T19:49:08.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.603+0000 7fea79e81640 10 monclient: monmap: 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:49:08.604649+0000 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:49:08.604649+0000 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:49:08.604 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.604+0000 7fea79e81640 1 Processor -- start 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.604+0000 7fea79e81640 1 -- start start 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.604+0000 7fea79e81640 10 monclient: init 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 10 monclient(hunting): picked mon.noname-a con 0x7fea740b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 10 monclient(hunting): picked mon.noname-b con 0x7fea740b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 10 monclient(hunting): picked mon.noname-c con 0x7fea7414be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740af940 con 0x7fea740b4d60 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740b0a70 con 0x7fea7414be90 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740b0d50 con 0x7fea740b1730 2024-03-25T19:49:08.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:08.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.605+0000 7fea79e81640 10 monclient(hunting): authenticate will time out at 5375.732466s 2024-03-25T19:49:08.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.606+0000 7fea737fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fea740b4d60 0x7fea7413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47750/0 (socket says 172.21.15.18:47750) 2024-03-25T19:49:08.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.606+0000 7fea737fe640 1 -- 172.21.15.18:0/1600046323 learned_addr learned my addr 172.21.15.18:0/1600046323 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.618+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223154131 0 0) 0x7fea740af940 con 0x7fea740b4d60 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.618+0000 7fea727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.618+0000 7fea727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.618+0000 7fea727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.618+0000 7fea727fc640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.618+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea5c003520 con 0x7fea740b4d60 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 108794247 0 0) 0x7fea5c003520 con 0x7fea740b4d60 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 >> v1:172.21.15.18:6790/0 conn(0x7fea740b1730 legacy=0x7fea740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 >> v1:172.21.15.53:6789/0 conn(0x7fea7414be90 legacy=0x7fea7414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 1 monclient: found mon.noname-a 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740b1030 con 0x7fea740b4d60 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:08.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea64001690 con 0x7fea740b4d60 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea727fc640 10 monclient: dump: 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:08.627 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:08.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea79e81640 5 monclient: authenticate success, global_id 4596 2024-03-25T19:49:08.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.625+0000 7fea79e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:08.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea64001860 con 0x7fea740b4d60 2024-03-25T19:49:08.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:08.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea727fc640 1 -- 172.21.15.18:0/1600046323 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea64001cb0 con 0x7fea740b4d60 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 4 set_mon_vals no callback set 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea727fc640 10 monclient: dump: 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:08.632 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 10 monclient: shutdown 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 1 -- 172.21.15.18:0/1600046323 >> v1:172.21.15.18:6789/0 conn(0x7fea740b4d60 legacy=0x7fea7413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 1 -- 172.21.15.18:0/1600046323 shutdown_connections 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 1 -- 172.21.15.18:0/1600046323 >> 172.21.15.18:0/1600046323 conn(0x7fea740617a0 msgr2=0x7fea74061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 1 -- 172.21.15.18:0/1600046323 shutdown_connections 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.631+0000 7fea79e81640 1 -- 172.21.15.18:0/1600046323 wait complete. 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.632+0000 7fea79e81640 10 monclient: build_initial_monmap 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.632+0000 7fea79e81640 10 monclient: monmap: 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:49:08.634 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.632+0000 7fea79e81640 1 librados: starting msgr at 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.632+0000 7fea79e81640 1 librados: starting objecter 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.632+0000 7fea79e81640 1 Processor -- start 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 1 -- start start 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 1 librados: setting wanted keys 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 1 librados: calling monclient init 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 10 monclient: init 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:08.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:08.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 10 monclient(hunting): picked mon.noname-b con 0x7fea740b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:08.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 10 monclient(hunting): picked mon.noname-a con 0x7fea740b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:08.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 10 monclient(hunting): picked mon.noname-c con 0x7fea7414be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:08.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea74135810 con 0x7fea740b4d60 2024-03-25T19:49:08.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7413a4c0 con 0x7fea740b1730 2024-03-25T19:49:08.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7413a7a0 con 0x7fea7414be90 2024-03-25T19:49:08.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:08.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.633+0000 7fea79e81640 10 monclient(hunting): authenticate will time out at 5375.760538s 2024-03-25T19:49:08.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.643+0000 7fea72ffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fea740b4d60 0x7fea741349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47782/0 (socket says 172.21.15.18:47782) 2024-03-25T19:49:08.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.643+0000 7fea72ffd640 1 -- 172.21.15.18:0/4045346528 learned_addr learned my addr 172.21.15.18:0/4045346528 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:08.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.645+0000 7fea5bfff640 10 client.?.objecter ms_handle_connect 0x7fea740b4d60 2024-03-25T19:49:08.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.645+0000 7fea5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.651+0000 7fea5bfff640 10 client.?.objecter ms_handle_connect 0x7fea7414be90 2024-03-25T19:49:08.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.651+0000 7fea5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.653+0000 7fea5bfff640 10 client.?.objecter ms_handle_connect 0x7fea740b1730 2024-03-25T19:49:08.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.653+0000 7fea5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.660+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785544747 0 0) 0x7fea7413a7a0 con 0x7fea7414be90 2024-03-25T19:49:08.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.660+0000 7fea5bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.660+0000 7fea5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.660+0000 7fea5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.660+0000 7fea5bfff640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:49:08.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.660+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea60004eb0 con 0x7fea7414be90 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.664+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935174039 0 0) 0x7fea7413a4c0 con 0x7fea740b1730 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.664+0000 7fea5bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.664+0000 7fea5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.664+0000 7fea5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.664+0000 7fea5bfff640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.664+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea7413a7a0 con 0x7fea740b1730 2024-03-25T19:49:08.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3192377175 0 0) 0x7fea7413a7a0 con 0x7fea740b1730 2024-03-25T19:49:08.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7413a4c0 con 0x7fea740b1730 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea64001b70 con 0x7fea740b1730 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 10 monclient(hunting): dump: 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:08.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1930047531 0 0) 0x7fea60004eb0 con 0x7fea7414be90 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea7413a7a0 con 0x7fea7414be90 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea6c001b90 con 0x7fea7414be90 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.682+0000 7fea5bfff640 10 monclient(hunting): dump: 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:08.684 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:08.685 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:08.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.688+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171804678 0 0) 0x7fea74135810 con 0x7fea740b4d60 2024-03-25T19:49:08.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.688+0000 7fea5bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.688+0000 7fea5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.688+0000 7fea5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.688+0000 7fea5bfff640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:49:08.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.688+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea60004eb0 con 0x7fea740b4d60 2024-03-25T19:49:08.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1478610195 0 0) 0x7fea7413a7a0 con 0x7fea7414be90 2024-03-25T19:49:08.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 >> v1:172.21.15.53:6789/0 conn(0x7fea740b1730 legacy=0x7fea741d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 >> v1:172.21.15.18:6789/0 conn(0x7fea740b4d60 legacy=0x7fea741349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:08.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 1 monclient: found mon.c 2024-03-25T19:49:08.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea741428f0 con 0x7fea7414be90 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 10 monclient: _finish_auth 0 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 5 monclient: authenticate success, global_id 4565 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 10 monclient: _renew_subs 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea741d8360 con 0x7fea7414be90 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 10 monclient: _renew_subs 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea741432d0 con 0x7fea7414be90 2024-03-25T19:49:08.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.691+0000 7fea79e81640 1 librados: init done 2024-03-25T19:49:08.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.692+0000 7fea79e81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:08.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.692+0000 7fea79e81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:08.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.692+0000 7fea79e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:08.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:08.692+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea741d7970 con 0x7fea7414be90 2024-03-25T19:49:09.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.010+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea6c002060 con 0x7fea7414be90 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.010+0000 7fea5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.010+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea6c003b10 con 0x7fea7414be90 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.010+0000 7fea5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.010+0000 7fea5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.010+0000 7fea5bfff640 10 monclient: dump: 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:09.011 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:09.012 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:09.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.010+0000 7fea717fa640 4 set_mon_vals no callback set 2024-03-25T19:49:09.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.011+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fea6c003d30 con 0x7fea7414be90 2024-03-25T19:49:09.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.012+0000 7fea737fe640 10 monclient: get_auth_request con 0x7fea6006dee0 auth_method 0 2024-03-25T19:49:09.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.012+0000 7fea5bfff640 10 client.4565.objecter ms_handle_connect 0x7fea6006dee0 2024-03-25T19:49:09.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.021+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fea6c053850 con 0x7fea7414be90 2024-03-25T19:49:09.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.021+0000 7fea5bfff640 10 client.4565.objecter ms_dispatch 0x7fea74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:09.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.021+0000 7fea5bfff640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:09.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.021+0000 7fea5bfff640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:09.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.021+0000 7fea5bfff640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-25T19:49:09.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.021+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea6c04f190 con 0x7fea7414be90 2024-03-25T19:49:09.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.021+0000 7fea5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:09.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.021+0000 7fea5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:09.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.270+0000 7fea79e81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.270+0000 7fea79e81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.270+0000 7fea79e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:09.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.270+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea740627b0 con 0x7fea7414be90 2024-03-25T19:49:09.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.293+0000 7fea5bfff640 1 -- 172.21.15.18:0/4045346528 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fea6c051290 con 0x7fea7414be90 2024-03-25T19:49:09.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.293+0000 7fea5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.293+0000 7fea5bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:09.293 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:49:09.293 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:09.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 10 librados: watch_flush enter 2024-03-25T19:49:09.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 10 librados: watch_flush exit 2024-03-25T19:49:09.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 10 client.4565.objecter successfully canceled tick 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 >> v1:172.21.15.18:6800/1520000573 conn(0x7fea6006dee0 legacy=0x7fea600703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 10 monclient: shutdown 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 >> v1:172.21.15.18:6790/0 conn(0x7fea7414be90 legacy=0x7fea74135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 shutdown_connections 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 >> 172.21.15.18:0/4045346528 conn(0x7fea740617a0 msgr2=0x7fea740b6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 shutdown_connections 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 1 -- 172.21.15.18:0/4045346528 wait complete. 2024-03-25T19:49:09.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:09.295+0000 7fea79e81640 1 librados: shutdown 2024-03-25T19:49:09.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:13.550 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:41.080 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:13.556 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:41.087 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:13.589 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:41.120 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:13.633 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:41.129 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:13.637 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:41.167 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:13.638 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:41.134 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:13.651 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:41.182 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:13.675 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:08:41.171 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:13.678 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:41.208 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:16.113 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:16.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.226+0000 7f07a0a84640 10 monclient: get_monmap_and_config 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.226+0000 7f07a0a84640 10 monclient: build_initial_monmap 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.226+0000 7f07a0a84640 10 monclient: monmap: 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:49:16.227707+0000 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:49:16.227707+0000 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:49:16.227 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:49:16.228 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:16.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 1 Processor -- start 2024-03-25T19:49:16.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 1 -- start start 2024-03-25T19:49:16.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 10 monclient: init 2024-03-25T19:49:16.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:16.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:16.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 10 monclient(hunting): picked mon.noname-a con 0x7f079c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:16.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 10 monclient(hunting): picked mon.noname-b con 0x7f079c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:16.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 10 monclient(hunting): picked mon.noname-c con 0x7f079c1430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:16.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c142520 con 0x7f079c0a98f0 2024-03-25T19:49:16.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c0b28a0 con 0x7f079c1430c0 2024-03-25T19:49:16.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c0b2a30 con 0x7f079c13f710 2024-03-25T19:49:16.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:16.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.228+0000 7f07a0a84640 10 monclient(hunting): authenticate will time out at 5383.355666s 2024-03-25T19:49:16.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.233+0000 7f079ad9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f079c1430c0 0x7f079c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:57866/0 (socket says 172.21.15.18:57866) 2024-03-25T19:49:16.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.233+0000 7f079ad9d640 1 -- 172.21.15.18:0/195050607 learned_addr learned my addr 172.21.15.18:0/195050607 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:16.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.240+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681309277 0 0) 0x7f079c0b28a0 con 0x7f079c1430c0 2024-03-25T19:49:16.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.240+0000 7f0799d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.240+0000 7f0799d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.240+0000 7f0799d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.240+0000 7f0799d9b640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:49:16.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.240+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0788003520 con 0x7f079c1430c0 2024-03-25T19:49:16.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.244+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319691986 0 0) 0x7f079c0b2a30 con 0x7f079c13f710 2024-03-25T19:49:16.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.244+0000 7f0799d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.244+0000 7f0799d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.244+0000 7f0799d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.244+0000 7f0799d9b640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:49:16.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.244+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f079c0b28a0 con 0x7f079c13f710 2024-03-25T19:49:16.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202367664 0 0) 0x7f0788003520 con 0x7f079c1430c0 2024-03-25T19:49:16.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 >> v1:172.21.15.18:6790/0 conn(0x7f079c13f710 legacy=0x7f079c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 >> v1:172.21.15.18:6789/0 conn(0x7f079c0a98f0 legacy=0x7f079c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:49:16.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:49:16.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 1 monclient: found mon.noname-c 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c0b44c0 con 0x7f079c1430c0 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 10 monclient: _finish_auth 0 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f0794001a10 con 0x7f079c1430c0 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f0799d9b640 10 monclient: dump: 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:16.252 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f07a0a84640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:49:16.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.251+0000 7f07a0a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.267+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0794001b70 con 0x7f079c1430c0 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.267+0000 7f0799d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.267+0000 7f0799d9b640 1 -- 172.21.15.18:0/195050607 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f0794001f80 con 0x7f079c1430c0 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.267+0000 7f0799d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.267+0000 7f0799d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.267+0000 7f0799d9b640 10 monclient: dump: 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.267 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:16.268 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:16.268 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:16.268 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:16.268 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:16.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.267+0000 7f07a0a84640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:16.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.267+0000 7f07a0a84640 4 set_mon_vals no callback set 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.268+0000 7f07a0a84640 10 monclient: shutdown 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.268+0000 7f07a0a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.268+0000 7f07a0a84640 1 -- 172.21.15.18:0/195050607 >> v1:172.21.15.53:6789/0 conn(0x7f079c1430c0 legacy=0x7f079c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.268+0000 7f07a0a84640 1 -- 172.21.15.18:0/195050607 shutdown_connections 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.268+0000 7f07a0a84640 1 -- 172.21.15.18:0/195050607 >> 172.21.15.18:0/195050607 conn(0x7f079c0b7ea0 msgr2=0x7f079c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.268+0000 7f07a0a84640 1 -- 172.21.15.18:0/195050607 shutdown_connections 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.268+0000 7f07a0a84640 1 -- 172.21.15.18:0/195050607 wait complete. 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.269+0000 7f07a0a84640 10 monclient: build_initial_monmap 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.269+0000 7f07a0a84640 10 monclient: monmap: 2024-03-25T19:49:16.269 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:49:16.270 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 1 librados: starting msgr at 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 1 librados: starting objecter 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 1 Processor -- start 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 1 -- start start 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 1 librados: setting wanted keys 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 1 librados: calling monclient init 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 10 monclient: init 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.270+0000 7f07a0a84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.271+0000 7f07a0a84640 10 monclient(hunting): picked mon.noname-b con 0x7f079c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.271+0000 7f07a0a84640 10 monclient(hunting): picked mon.noname-a con 0x7f079c13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.271+0000 7f07a0a84640 10 monclient(hunting): picked mon.noname-c con 0x7f079c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:16.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.271+0000 7f07a0a84640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c0abae0 con 0x7f079c13f710 2024-03-25T19:49:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.271+0000 7f07a0a84640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c0acb60 con 0x7f079c0a98f0 2024-03-25T19:49:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.271+0000 7f07a0a84640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c0abd50 con 0x7f079c1430c0 2024-03-25T19:49:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.271+0000 7f07a0a84640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.271+0000 7f07a0a84640 10 monclient(hunting): authenticate will time out at 5383.398205s 2024-03-25T19:49:16.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.275+0000 7f079a59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f079c0a98f0 0x7f079c138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:57898/0 (socket says 172.21.15.18:57898) 2024-03-25T19:49:16.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.275+0000 7f079a59c640 1 -- 172.21.15.18:0/2508469838 learned_addr learned my addr 172.21.15.18:0/2508469838 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:16.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.276+0000 7f0792ffd640 10 client.?.objecter ms_handle_connect 0x7f079c0a98f0 2024-03-25T19:49:16.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.276+0000 7f0792ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.285+0000 7f0792ffd640 10 client.?.objecter ms_handle_connect 0x7f079c13f710 2024-03-25T19:49:16.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.285+0000 7f0792ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.288+0000 7f0792ffd640 10 client.?.objecter ms_handle_connect 0x7f079c1430c0 2024-03-25T19:49:16.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.288+0000 7f0792ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.292+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642230075 0 0) 0x7f079c0acb60 con 0x7f079c0a98f0 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.292+0000 7f0792ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.292+0000 7f0792ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.292+0000 7f0792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.292+0000 7f0792ffd640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:49:16.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.292+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f077c004eb0 con 0x7f079c0a98f0 2024-03-25T19:49:16.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.301+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1599448536 0 0) 0x7f077c004eb0 con 0x7f079c0a98f0 2024-03-25T19:49:16.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.301+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f079c0acb60 con 0x7f079c0a98f0 2024-03-25T19:49:16.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.301+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f078c004080 con 0x7f079c0a98f0 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.301+0000 7f0792ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.301+0000 7f0792ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.301+0000 7f0792ffd640 10 monclient(hunting): dump: 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:16.302 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:16.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.301+0000 7f0792ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:16.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.309+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307480701 0 0) 0x7f079c0abd50 con 0x7f079c1430c0 2024-03-25T19:49:16.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.309+0000 7f0792ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.309+0000 7f0792ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.309+0000 7f0792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.309+0000 7f0792ffd640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:49:16.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.309+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f077c004eb0 con 0x7f079c1430c0 2024-03-25T19:49:16.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3035688669 0 0) 0x7f079c0acb60 con 0x7f079c0a98f0 2024-03-25T19:49:16.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 >> v1:172.21.15.18:6790/0 conn(0x7f079c1430c0 legacy=0x7f079c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 >> v1:172.21.15.18:6789/0 conn(0x7f079c13f710 legacy=0x7f079c0ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 1 monclient: found mon.b 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c0ac030 con 0x7f079c0a98f0 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f0792ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 5 monclient: authenticate success, global_id 4573 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 10 monclient: _renew_subs 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c1390d0 con 0x7f079c0a98f0 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:16.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 10 monclient: _renew_subs 2024-03-25T19:49:16.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:16.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f079c136520 con 0x7f079c0a98f0 2024-03-25T19:49:16.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.313+0000 7f07a0a84640 1 librados: init done 2024-03-25T19:49:16.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.314+0000 7f07a0a84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.314+0000 7f07a0a84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.314+0000 7f07a0a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:16.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.314+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079c0627b0 con 0x7f079c0a98f0 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.596+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f078c000ed0 con 0x7f079c0a98f0 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.596+0000 7f0792ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.596+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f078c001260 con 0x7f079c0a98f0 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.596+0000 7f0792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.596+0000 7f0792ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.596+0000 7f0792ffd640 10 monclient: dump: 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:16.596 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.597 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:16.597 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:16.597 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:16.597 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:16.597 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:16.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.596+0000 7f079959a640 4 set_mon_vals no callback set 2024-03-25T19:49:16.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.597+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f078c001480 con 0x7f079c0a98f0 2024-03-25T19:49:16.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.597+0000 7f0793fff640 10 monclient: get_auth_request con 0x7f077c06de20 auth_method 0 2024-03-25T19:49:16.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.598+0000 7f0792ffd640 10 client.4573.objecter ms_handle_connect 0x7f077c06de20 2024-03-25T19:49:16.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.602+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f078c054780 con 0x7f079c0a98f0 2024-03-25T19:49:16.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.602+0000 7f0792ffd640 10 client.4573.objecter ms_dispatch 0x7f079c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:16.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.602+0000 7f0792ffd640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:16.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.602+0000 7f0792ffd640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:16.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.602+0000 7f0792ffd640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-25T19:49:16.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.602+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f078c050290 con 0x7f079c0a98f0 2024-03-25T19:49:16.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.602+0000 7f0792ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.602+0000 7f0792ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:16.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.837+0000 7f07a0a84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:16.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.837+0000 7f07a0a84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:16.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.837+0000 7f07a0a84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:16.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.837+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f079c13ce70 con 0x7f079c0a98f0 2024-03-25T19:49:16.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.904+0000 7f0792ffd640 1 -- 172.21.15.18:0/2508469838 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f078c0521c0 con 0x7f079c0a98f0 2024-03-25T19:49:16.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.904+0000 7f0792ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:16.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.904+0000 7f0792ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:16.904 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:49:16.904 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 10 librados: watch_flush enter 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 10 librados: watch_flush exit 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 10 client.4573.objecter successfully canceled tick 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 >> v1:172.21.15.18:6800/1520000573 conn(0x7f077c06de20 legacy=0x7f077c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 10 monclient: shutdown 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 >> v1:172.21.15.53:6789/0 conn(0x7f079c0a98f0 legacy=0x7f079c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 shutdown_connections 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.906+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 >> 172.21.15.18:0/2508469838 conn(0x7f079c0b7ea0 msgr2=0x7f079c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.907+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 shutdown_connections 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.907+0000 7f07a0a84640 1 -- 172.21.15.18:0/2508469838 wait complete. 2024-03-25T19:49:16.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:16.907+0000 7f07a0a84640 1 librados: shutdown 2024-03-25T19:49:16.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:23.725 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:23.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.840+0000 7fea5aa9a640 10 monclient: get_monmap_and_config 2024-03-25T19:49:23.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.840+0000 7fea5aa9a640 10 monclient: build_initial_monmap 2024-03-25T19:49:23.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.840+0000 7fea5aa9a640 10 monclient: monmap: 2024-03-25T19:49:23.840 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:23.840 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:23.840 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:49:23.841299+0000 2024-03-25T19:49:23.840 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:49:23.841299+0000 2024-03-25T19:49:23.841 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:23.841 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:23.841 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:23.841 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:49:23.841 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:49:23.841 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:23.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.841+0000 7fea5aa9a640 1 Processor -- start 2024-03-25T19:49:23.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.841+0000 7fea5aa9a640 1 -- start start 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.841+0000 7fea5aa9a640 10 monclient: init 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.841+0000 7fea5aa9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.841+0000 7fea5aa9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.841+0000 7fea5aa9a640 10 monclient(hunting): picked mon.noname-a con 0x7fea540a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.841+0000 7fea5aa9a640 10 monclient(hunting): picked mon.noname-c con 0x7fea5413f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.842+0000 7fea5aa9a640 10 monclient(hunting): picked mon.noname-b con 0x7fea541430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.842+0000 7fea5aa9a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea54142520 con 0x7fea540a98f0 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.842+0000 7fea5aa9a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea540b28a0 con 0x7fea5413f710 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.842+0000 7fea5aa9a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea540b2a30 con 0x7fea541430c0 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.842+0000 7fea5aa9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.842+0000 7fea5aa9a640 10 monclient(hunting): authenticate will time out at 5390.968797s 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.842+0000 7fea59037640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fea541430c0 0x7fea5414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40818/0 (socket says 172.21.15.18:40818) 2024-03-25T19:49:23.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.842+0000 7fea59037640 1 -- 172.21.15.18:0/3702688900 learned_addr learned my addr 172.21.15.18:0/3702688900 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:23.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.901+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585642539 0 0) 0x7fea54142520 con 0x7fea540a98f0 2024-03-25T19:49:23.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.901+0000 7fea4b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:23.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.901+0000 7fea4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:23.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.901+0000 7fea4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:23.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.901+0000 7fea4b7fe640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:49:23.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.901+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c003520 con 0x7fea540a98f0 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544445396 0 0) 0x7fea3c003520 con 0x7fea540a98f0 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 >> v1:172.21.15.18:6790/0 conn(0x7fea541430c0 legacy=0x7fea5414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 >> v1:172.21.15.53:6789/0 conn(0x7fea5413f710 legacy=0x7fea541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea540b44c0 con 0x7fea540a98f0 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea50001690 con 0x7fea540a98f0 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:23.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea4b7fe640 10 monclient: dump: 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea5aa9a640 5 monclient: authenticate success, global_id 4605 2024-03-25T19:49:23.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.916+0000 7fea5aa9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:23.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.938+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea500017f0 con 0x7fea540a98f0 2024-03-25T19:49:23.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.938+0000 7fea4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:23.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.938+0000 7fea4b7fe640 1 -- 172.21.15.18:0/3702688900 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea50001c00 con 0x7fea540a98f0 2024-03-25T19:49:23.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.938+0000 7fea4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:23.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.938+0000 7fea4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:23.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.938+0000 7fea4b7fe640 10 monclient: dump: 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.938+0000 7fea5aa9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:23.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.938+0000 7fea5aa9a640 4 set_mon_vals no callback set 2024-03-25T19:49:23.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.939+0000 7fea5aa9a640 10 monclient: shutdown 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.939+0000 7fea5aa9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.939+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3702688900 >> v1:172.21.15.18:6789/0 conn(0x7fea540a98f0 legacy=0x7fea540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.939+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3702688900 shutdown_connections 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.939+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3702688900 >> 172.21.15.18:0/3702688900 conn(0x7fea540b7ea0 msgr2=0x7fea540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.939+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3702688900 shutdown_connections 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.939+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3702688900 wait complete. 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.940+0000 7fea5aa9a640 10 monclient: build_initial_monmap 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.940+0000 7fea5aa9a640 10 monclient: monmap: 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:23.941 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.940+0000 7fea5aa9a640 1 librados: starting msgr at 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.941+0000 7fea5aa9a640 1 librados: starting objecter 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.941+0000 7fea5aa9a640 1 Processor -- start 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.941+0000 7fea5aa9a640 1 -- start start 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.941+0000 7fea5aa9a640 1 librados: setting wanted keys 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.941+0000 7fea5aa9a640 1 librados: calling monclient init 2024-03-25T19:49:23.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.941+0000 7fea5aa9a640 10 monclient: init 2024-03-25T19:49:23.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:23.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:23.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 10 monclient(hunting): picked mon.noname-c con 0x7fea540a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:23.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 10 monclient(hunting): picked mon.noname-a con 0x7fea5413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:23.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 10 monclient(hunting): picked mon.noname-b con 0x7fea541430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:23.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea54138c70 con 0x7fea5413f710 2024-03-25T19:49:23.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea541367d0 con 0x7fea541430c0 2024-03-25T19:49:23.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea54136960 con 0x7fea540a98f0 2024-03-25T19:49:23.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:23.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.942+0000 7fea5aa9a640 10 monclient(hunting): authenticate will time out at 5391.069148s 2024-03-25T19:49:23.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.958+0000 7fea59037640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fea541430c0 0x7fea541e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:34708/0 (socket says 172.21.15.18:34708) 2024-03-25T19:49:23.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.958+0000 7fea59037640 1 -- 172.21.15.18:0/3685983767 learned_addr learned my addr 172.21.15.18:0/3685983767 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:23.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.960+0000 7fea497fa640 10 client.?.objecter ms_handle_connect 0x7fea541430c0 2024-03-25T19:49:23.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.960+0000 7fea497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:23.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.961+0000 7fea497fa640 10 client.?.objecter ms_handle_connect 0x7fea5413f710 2024-03-25T19:49:23.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.961+0000 7fea497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:23.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.968+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875786464 0 0) 0x7fea541367d0 con 0x7fea541430c0 2024-03-25T19:49:23.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.968+0000 7fea497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:23.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.968+0000 7fea497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:23.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.968+0000 7fea497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:23.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.968+0000 7fea497fa640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:49:23.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.968+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea40004eb0 con 0x7fea541430c0 2024-03-25T19:49:23.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.970+0000 7fea497fa640 10 client.?.objecter ms_handle_connect 0x7fea540a98f0 2024-03-25T19:49:23.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.970+0000 7fea497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:23.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.976+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388374057 0 0) 0x7fea40004eb0 con 0x7fea541430c0 2024-03-25T19:49:23.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.976+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea541367d0 con 0x7fea541430c0 2024-03-25T19:49:23.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.976+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea4c001b10 con 0x7fea541430c0 2024-03-25T19:49:23.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.976+0000 7fea497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.976+0000 7fea497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.976+0000 7fea497fa640 10 monclient(hunting): dump: 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:23.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.976+0000 7fea497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785654301 0 0) 0x7fea54138c70 con 0x7fea5413f710 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea40004eb0 con 0x7fea5413f710 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639228011 0 0) 0x7fea54136960 con 0x7fea540a98f0 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:23.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:23.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:23.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:49:23.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.981+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea54138c70 con 0x7fea540a98f0 2024-03-25T19:49:23.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1889547655 0 0) 0x7fea541367d0 con 0x7fea541430c0 2024-03-25T19:49:23.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 >> v1:172.21.15.18:6790/0 conn(0x7fea540a98f0 legacy=0x7fea54137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:23.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 >> v1:172.21.15.18:6789/0 conn(0x7fea5413f710 legacy=0x7fea54132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:23.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 10 monclient: _finish_hunting 0 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 1 monclient: found mon.b 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea54139630 con 0x7fea541430c0 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 10 monclient: _finish_auth 0 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 10 monclient: _check_auth_tickets 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 10 monclient: _renew_subs 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea54139880 con 0x7fea541430c0 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 10 monclient: _renew_subs 2024-03-25T19:49:23.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:23.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea541337f0 con 0x7fea541430c0 2024-03-25T19:49:23.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.987+0000 7fea5aa9a640 1 librados: init done 2024-03-25T19:49:23.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.989+0000 7fea5aa9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:23.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.989+0000 7fea5aa9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:23.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.989+0000 7fea5aa9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:23.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:23.989+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea540627b0 con 0x7fea541430c0 2024-03-25T19:49:24.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.205+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea4c002040 con 0x7fea541430c0 2024-03-25T19:49:24.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.205+0000 7fea497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:24.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.205+0000 7fea4affd640 4 set_mon_vals no callback set 2024-03-25T19:49:24.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.280+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fea4c004d00 con 0x7fea541430c0 2024-03-25T19:49:24.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.280+0000 7fea497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.280+0000 7fea497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:24.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.280+0000 7fea497fa640 10 monclient: dump: 2024-03-25T19:49:24.280 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:24.281 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:24.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.281+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fea4c004ee0 con 0x7fea541430c0 2024-03-25T19:49:24.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.282+0000 7fea58836640 10 monclient: get_auth_request con 0x7fea4006de80 auth_method 0 2024-03-25T19:49:24.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.282+0000 7fea497fa640 10 client.4579.objecter ms_handle_connect 0x7fea4006de80 2024-03-25T19:49:24.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.286+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fea4c054b10 con 0x7fea541430c0 2024-03-25T19:49:24.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.286+0000 7fea497fa640 10 client.4579.objecter ms_dispatch 0x7fea54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:24.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.286+0000 7fea497fa640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:24.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.286+0000 7fea497fa640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:24.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.286+0000 7fea497fa640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.289+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea4c050450 con 0x7fea541430c0 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.289+0000 7fea497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.289+0000 7fea497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:24.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.545+0000 7fea5aa9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.545+0000 7fea5aa9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.545+0000 7fea5aa9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:24.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.545+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea54069f40 con 0x7fea541430c0 2024-03-25T19:49:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.564+0000 7fea497fa640 1 -- 172.21.15.18:0/3685983767 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fea4c052550 con 0x7fea541430c0 2024-03-25T19:49:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.564+0000 7fea497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.564+0000 7fea497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:24.565 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:49:24.565 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 10 librados: watch_flush enter 2024-03-25T19:49:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 10 librados: watch_flush exit 2024-03-25T19:49:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T19:49:24.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 10 client.4579.objecter successfully canceled tick 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 >> v1:172.21.15.18:6800/1520000573 conn(0x7fea4006de80 legacy=0x7fea40070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 10 monclient: shutdown 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 >> v1:172.21.15.53:6789/0 conn(0x7fea541430c0 legacy=0x7fea541e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 shutdown_connections 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 >> 172.21.15.18:0/3685983767 conn(0x7fea540b7ea0 msgr2=0x7fea54074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 shutdown_connections 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 1 -- 172.21.15.18:0/3685983767 wait complete. 2024-03-25T19:49:24.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:24.567+0000 7fea5aa9a640 1 librados: shutdown 2024-03-25T19:49:24.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:31.385 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:31.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.500+0000 7ff9ef5e8640 10 monclient: get_monmap_and_config 2024-03-25T19:49:31.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.500+0000 7ff9ef5e8640 10 monclient: build_initial_monmap 2024-03-25T19:49:31.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.500+0000 7ff9ef5e8640 10 monclient: monmap: 2024-03-25T19:49:31.500 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:31.501 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:31.501 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:49:31.501499+0000 2024-03-25T19:49:31.501 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:49:31.501499+0000 2024-03-25T19:49:31.501 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:31.501 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:31.501 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:31.501 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:49:31.501 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:49:31.502 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:31.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.501+0000 7ff9ef5e8640 1 Processor -- start 2024-03-25T19:49:31.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.501+0000 7ff9ef5e8640 1 -- start start 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.501+0000 7ff9ef5e8640 10 monclient: init 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff9e80b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e80b1730 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e814be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e80af940 con 0x7ff9e814be90 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e80b0a70 con 0x7ff9e80b1730 2024-03-25T19:49:31.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e80b0d50 con 0x7ff9e80b4d60 2024-03-25T19:49:31.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:31.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.502+0000 7ff9ef5e8640 10 monclient(hunting): authenticate will time out at 5398.629567s 2024-03-25T19:49:31.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.512+0000 7ff9ed384640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ff9e80b4d60 0x7ff9e813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57178/0 (socket says 172.21.15.18:57178) 2024-03-25T19:49:31.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.512+0000 7ff9ed384640 1 -- 172.21.15.18:0/3014829966 learned_addr learned my addr 172.21.15.18:0/3014829966 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:31.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.528+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849062021 0 0) 0x7ff9e80b0d50 con 0x7ff9e80b4d60 2024-03-25T19:49:31.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.528+0000 7ff9dffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.528+0000 7ff9dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.528+0000 7ff9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.528+0000 7ff9dffff640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:49:31.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.528+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9d4003520 con 0x7ff9e80b4d60 2024-03-25T19:49:31.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.535+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141135131 0 0) 0x7ff9e80b0a70 con 0x7ff9e80b1730 2024-03-25T19:49:31.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.535+0000 7ff9dffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.535+0000 7ff9dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.535+0000 7ff9dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.535+0000 7ff9dffff640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:49:31.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.535+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e80b0d50 con 0x7ff9e80b1730 2024-03-25T19:49:31.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.541+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3606152626 0 0) 0x7ff9d4003520 con 0x7ff9e80b4d60 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.541+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 >> v1:172.21.15.53:6789/0 conn(0x7ff9e80b1730 legacy=0x7ff9e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.541+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 >> v1:172.21.15.18:6789/0 conn(0x7ff9e814be90 legacy=0x7ff9e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 1 monclient: found mon.noname-b 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e80b1030 con 0x7ff9e80b4d60 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 10 monclient: _finish_auth 0 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff9e4001690 con 0x7ff9e80b4d60 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9ef5e8640 5 monclient: authenticate success, global_id 4580 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9ef5e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:31.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9dffff640 10 monclient: dump: 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:31.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.542+0000 7ff9ef5e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:31.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.563+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9e40017f0 con 0x7ff9e80b4d60 2024-03-25T19:49:31.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.563+0000 7ff9dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:31.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.563+0000 7ff9dffff640 1 -- 172.21.15.18:0/3014829966 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff9e4001c00 con 0x7ff9e80b4d60 2024-03-25T19:49:31.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.563+0000 7ff9ef5e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:31.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.563+0000 7ff9ef5e8640 4 set_mon_vals no callback set 2024-03-25T19:49:31.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.563+0000 7ff9dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.563+0000 7ff9dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.563+0000 7ff9dffff640 10 monclient: dump: 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:31.564 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:31.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.564+0000 7ff9ef5e8640 10 monclient: shutdown 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.564+0000 7ff9ef5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.564+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/3014829966 >> v1:172.21.15.18:6790/0 conn(0x7ff9e80b4d60 legacy=0x7ff9e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.564+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/3014829966 shutdown_connections 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.564+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/3014829966 >> 172.21.15.18:0/3014829966 conn(0x7ff9e80617a0 msgr2=0x7ff9e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.564+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/3014829966 shutdown_connections 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.564+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/3014829966 wait complete. 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.565+0000 7ff9ef5e8640 10 monclient: build_initial_monmap 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.565+0000 7ff9ef5e8640 10 monclient: monmap: 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:49:31.566 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.565+0000 7ff9ef5e8640 1 librados: starting msgr at 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.565+0000 7ff9ef5e8640 1 librados: starting objecter 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 1 Processor -- start 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 1 -- start start 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 1 librados: setting wanted keys 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 1 librados: calling monclient init 2024-03-25T19:49:31.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 10 monclient: init 2024-03-25T19:49:31.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:31.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:31.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 10 monclient(hunting): picked mon.noname-c con 0x7ff9e80b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:31.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 10 monclient(hunting): picked mon.noname-a con 0x7ff9e80b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:31.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 10 monclient(hunting): picked mon.noname-b con 0x7ff9e814be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:31.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e81361d0 con 0x7ff9e80b4d60 2024-03-25T19:49:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e8136c40 con 0x7ff9e814be90 2024-03-25T19:49:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e8136f20 con 0x7ff9e80b1730 2024-03-25T19:49:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.566+0000 7ff9ef5e8640 10 monclient(hunting): authenticate will time out at 5398.693685s 2024-03-25T19:49:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.571+0000 7ff9edb85640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff9e814be90 0x7ff9e81e33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:52372/0 (socket says 172.21.15.18:52372) 2024-03-25T19:49:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.571+0000 7ff9edb85640 1 -- 172.21.15.18:0/620706824 learned_addr learned my addr 172.21.15.18:0/620706824 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:31.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.574+0000 7ff9ddffb640 10 client.?.objecter ms_handle_connect 0x7ff9e814be90 2024-03-25T19:49:31.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.574+0000 7ff9ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.575+0000 7ff9ddffb640 10 client.?.objecter ms_handle_connect 0x7ff9e80b4d60 2024-03-25T19:49:31.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.575+0000 7ff9ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.590+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285742757 0 0) 0x7ff9e8136c40 con 0x7ff9e814be90 2024-03-25T19:49:31.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.590+0000 7ff9ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.590+0000 7ff9ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.590+0000 7ff9ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.590+0000 7ff9ddffb640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:49:31.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.590+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9cc004eb0 con 0x7ff9e814be90 2024-03-25T19:49:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.593+0000 7ff9ddffb640 10 client.?.objecter ms_handle_connect 0x7ff9e80b1730 2024-03-25T19:49:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.593+0000 7ff9ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.593+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163699700 0 0) 0x7ff9e81361d0 con 0x7ff9e80b4d60 2024-03-25T19:49:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.593+0000 7ff9ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.593+0000 7ff9ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.593+0000 7ff9ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.593+0000 7ff9ddffb640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:49:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.593+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e8136c40 con 0x7ff9e80b4d60 2024-03-25T19:49:31.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.600+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450014957 0 0) 0x7ff9e8136f20 con 0x7ff9e80b1730 2024-03-25T19:49:31.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.600+0000 7ff9ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.600+0000 7ff9ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.600+0000 7ff9ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.600+0000 7ff9ddffb640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:49:31.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.600+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e81361d0 con 0x7ff9e80b1730 2024-03-25T19:49:31.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.607+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3197415878 0 0) 0x7ff9cc004eb0 con 0x7ff9e814be90 2024-03-25T19:49:31.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.607+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9e8136f20 con 0x7ff9e814be90 2024-03-25T19:49:31.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.607+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff9e0001c30 con 0x7ff9e814be90 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.607+0000 7ff9ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.607+0000 7ff9ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.607+0000 7ff9ddffb640 10 monclient(hunting): dump: 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:31.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.608+0000 7ff9ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:31.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.613+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1851141029 0 0) 0x7ff9e81361d0 con 0x7ff9e80b1730 2024-03-25T19:49:31.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.613+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9cc004eb0 con 0x7ff9e80b1730 2024-03-25T19:49:31.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.613+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff9e4001b00 con 0x7ff9e80b1730 2024-03-25T19:49:31.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.613+0000 7ff9ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.613+0000 7ff9ddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.613+0000 7ff9ddffb640 10 monclient(hunting): dump: 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:31.614 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1158061172 0 0) 0x7ff9e8136f20 con 0x7ff9e814be90 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 >> v1:172.21.15.18:6790/0 conn(0x7ff9e80b1730 legacy=0x7ff9e812e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 >> v1:172.21.15.18:6789/0 conn(0x7ff9e80b4d60 legacy=0x7ff9e812cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 1 monclient: found mon.b 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e8137200 con 0x7ff9e814be90 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 10 monclient: _renew_subs 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8135cb0 con 0x7ff9e814be90 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 10 monclient: _renew_subs 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e813e930 con 0x7ff9e814be90 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.616+0000 7ff9ef5e8640 1 librados: init done 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.617+0000 7ff9ef5e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.617+0000 7ff9ef5e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.617+0000 7ff9ef5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.617+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e81401f0 con 0x7ff9e814be90 2024-03-25T19:49:31.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.910+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9e0003990 con 0x7ff9e814be90 2024-03-25T19:49:31.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.910+0000 7ff9ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:31.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.910+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff9e0003da0 con 0x7ff9e814be90 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.910+0000 7ff9ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.910+0000 7ff9ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.910+0000 7ff9ddffb640 10 monclient: dump: 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.910+0000 7ff9df7fe640 4 set_mon_vals no callback set 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.911+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ff9e0003fc0 con 0x7ff9e814be90 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.912+0000 7ff9ed384640 10 monclient: get_auth_request con 0x7ff9cc06df90 auth_method 0 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.912+0000 7ff9ddffb640 10 client.4585.objecter ms_handle_connect 0x7ff9cc06df90 2024-03-25T19:49:31.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.921+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7ff9e00549c0 con 0x7ff9e814be90 2024-03-25T19:49:31.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.921+0000 7ff9ddffb640 10 client.4585.objecter ms_dispatch 0x7ff9e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:31.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.921+0000 7ff9ddffb640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:31.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.921+0000 7ff9ddffb640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:31.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.921+0000 7ff9ddffb640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T19:49:31.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.921+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9e00504d0 con 0x7ff9e814be90 2024-03-25T19:49:31.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.921+0000 7ff9ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:31.921+0000 7ff9ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:32.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.156+0000 7ff9ef5e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.156+0000 7ff9ef5e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.156+0000 7ff9ef5e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:32.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.156+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9e80afbd0 con 0x7ff9e814be90 2024-03-25T19:49:32.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.305+0000 7ff9ddffb640 1 -- 172.21.15.18:0/620706824 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7ff9e0052400 con 0x7ff9e814be90 2024-03-25T19:49:32.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.305+0000 7ff9ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.305+0000 7ff9ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:32.306 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:49:32.306 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 10 librados: watch_flush enter 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 10 librados: watch_flush exit 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 10 client.4585.objecter successfully canceled tick 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 >> v1:172.21.15.18:6800/1520000573 conn(0x7ff9cc06df90 legacy=0x7ff9cc070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 10 monclient: shutdown 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 >> v1:172.21.15.53:6789/0 conn(0x7ff9e814be90 legacy=0x7ff9e81e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 shutdown_connections 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.308+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 >> 172.21.15.18:0/620706824 conn(0x7ff9e80617a0 msgr2=0x7ff9e80b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.309+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 shutdown_connections 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.309+0000 7ff9ef5e8640 1 -- 172.21.15.18:0/620706824 wait complete. 2024-03-25T19:49:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:32.309+0000 7ff9ef5e8640 1 librados: shutdown 2024-03-25T19:49:32.320 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:33.651 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:01.182 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.663 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:01.194 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.700 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:01.230 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.726 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:01.222 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.728 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:01.259 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.732 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:01.229 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.733 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:01.264 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.762 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:01.258 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:33.767 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:01.298 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:39.127 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.244+0000 7f6a85d4d640 10 monclient: get_monmap_and_config 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.244+0000 7f6a85d4d640 10 monclient: build_initial_monmap 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.244+0000 7f6a85d4d640 10 monclient: monmap: 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:49:39.245213+0000 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:49:39.245213+0000 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:39.244 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:49:39.245 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:49:39.245 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.245+0000 7f6a85d4d640 1 Processor -- start 2024-03-25T19:49:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.245+0000 7f6a85d4d640 1 -- start start 2024-03-25T19:49:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.245+0000 7f6a85d4d640 10 monclient: init 2024-03-25T19:49:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.245+0000 7f6a85d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.245+0000 7f6a85d4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.245+0000 7f6a85d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a800b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.245+0000 7f6a85d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a800b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.245+0000 7f6a85d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a8014be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.246+0000 7f6a85d4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a800af940 con 0x7f6a8014be90 2024-03-25T19:49:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.246+0000 7f6a85d4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a800b0a70 con 0x7f6a800b4d60 2024-03-25T19:49:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.246+0000 7f6a85d4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a800b0d50 con 0x7f6a800b1730 2024-03-25T19:49:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.246+0000 7f6a85d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.246+0000 7f6a85d4d640 10 monclient(hunting): authenticate will time out at 5406.372787s 2024-03-25T19:49:39.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.259+0000 7f6a7f7fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6a800b4d60 0x7f6a8013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:33664/0 (socket says 172.21.15.18:33664) 2024-03-25T19:49:39.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.259+0000 7f6a7f7fe640 1 -- 172.21.15.18:0/774712098 learned_addr learned my addr 172.21.15.18:0/774712098 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:39.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.271+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131810446 0 0) 0x7f6a800b0d50 con 0x7f6a800b1730 2024-03-25T19:49:39.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.271+0000 7f6a7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.271+0000 7f6a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.271+0000 7f6a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.271+0000 7f6a7e7fc640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:49:39.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.271+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a68003520 con 0x7f6a800b1730 2024-03-25T19:49:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.274+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020989188 0 0) 0x7f6a800b0a70 con 0x7f6a800b4d60 2024-03-25T19:49:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.274+0000 7f6a7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.274+0000 7f6a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.274+0000 7f6a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.274+0000 7f6a7e7fc640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:49:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.274+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a800b0d50 con 0x7f6a800b4d60 2024-03-25T19:49:39.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2144696880 0 0) 0x7f6a68003520 con 0x7f6a800b1730 2024-03-25T19:49:39.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 >> v1:172.21.15.53:6789/0 conn(0x7f6a800b4d60 legacy=0x7f6a8013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 >> v1:172.21.15.18:6789/0 conn(0x7f6a8014be90 legacy=0x7f6a8014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:39.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:49:39.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a800b1030 con 0x7f6a800b1730 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a85d4d640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6a74001690 con 0x7f6a800b1730 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a7e7fc640 10 monclient: dump: 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.280 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.281 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:39.281 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:39.281 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:39.281 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:39.281 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:39.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.279+0000 7f6a85d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:39.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.288+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a740017f0 con 0x7f6a800b1730 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.288+0000 7f6a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.288+0000 7f6a7e7fc640 1 -- 172.21.15.18:0/774712098 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6a74001c00 con 0x7f6a800b1730 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.288+0000 7f6a85d4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.288+0000 7f6a85d4d640 4 set_mon_vals no callback set 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.288+0000 7f6a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.288+0000 7f6a7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.288+0000 7f6a7e7fc640 10 monclient: dump: 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:39.288 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:39.289 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:39.289 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:39.289 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 10 monclient: shutdown 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 1 -- 172.21.15.18:0/774712098 >> v1:172.21.15.18:6790/0 conn(0x7f6a800b1730 legacy=0x7f6a800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 1 -- 172.21.15.18:0/774712098 shutdown_connections 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 1 -- 172.21.15.18:0/774712098 >> 172.21.15.18:0/774712098 conn(0x7f6a800617a0 msgr2=0x7f6a80061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 1 -- 172.21.15.18:0/774712098 shutdown_connections 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 1 -- 172.21.15.18:0/774712098 wait complete. 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 10 monclient: build_initial_monmap 2024-03-25T19:49:39.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.289+0000 7f6a85d4d640 10 monclient: monmap: 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.290+0000 7f6a85d4d640 1 librados: starting msgr at 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.290+0000 7f6a85d4d640 1 librados: starting objecter 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.290+0000 7f6a85d4d640 1 Processor -- start 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.290+0000 7f6a85d4d640 1 -- start start 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.290+0000 7f6a85d4d640 1 librados: setting wanted keys 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.290+0000 7f6a85d4d640 1 librados: calling monclient init 2024-03-25T19:49:39.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.290+0000 7f6a85d4d640 10 monclient: init 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a800b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a800b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a8014be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a80136250 con 0x7f6a800b4d60 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a80136af0 con 0x7f6a8014be90 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a80136c80 con 0x7f6a800b1730 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:39.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.291+0000 7f6a85d4d640 10 monclient(hunting): authenticate will time out at 5406.418023s 2024-03-25T19:49:39.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.304+0000 7f6a7ffff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6a8014be90 0x7f6a801e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:33684/0 (socket says 172.21.15.18:33684) 2024-03-25T19:49:39.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.304+0000 7f6a7ffff640 1 -- 172.21.15.18:0/1802180403 learned_addr learned my addr 172.21.15.18:0/1802180403 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:39.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.309+0000 7f6a67fff640 10 client.?.objecter ms_handle_connect 0x7f6a8014be90 2024-03-25T19:49:39.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.309+0000 7f6a67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.314+0000 7f6a67fff640 10 client.?.objecter ms_handle_connect 0x7f6a800b4d60 2024-03-25T19:49:39.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.314+0000 7f6a67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.314+0000 7f6a67fff640 10 client.?.objecter ms_handle_connect 0x7f6a800b1730 2024-03-25T19:49:39.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.314+0000 7f6a67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.327+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37973001 0 0) 0x7f6a80136250 con 0x7f6a800b4d60 2024-03-25T19:49:39.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.327+0000 7f6a67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.327+0000 7f6a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.327+0000 7f6a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.327+0000 7f6a67fff640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:49:39.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.327+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a6c004eb0 con 0x7f6a800b4d60 2024-03-25T19:49:39.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.335+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437503259 0 0) 0x7f6a80136af0 con 0x7f6a8014be90 2024-03-25T19:49:39.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.335+0000 7f6a67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.335+0000 7f6a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.335+0000 7f6a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.335+0000 7f6a67fff640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:49:39.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.335+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a80136250 con 0x7f6a8014be90 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.336+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572656783 0 0) 0x7f6a80136c80 con 0x7f6a800b1730 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.336+0000 7f6a67fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.336+0000 7f6a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.336+0000 7f6a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.336+0000 7f6a67fff640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.336+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a80136af0 con 0x7f6a800b1730 2024-03-25T19:49:39.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.345+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445734139 0 0) 0x7f6a80136af0 con 0x7f6a800b1730 2024-03-25T19:49:39.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.345+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a80136c80 con 0x7f6a800b1730 2024-03-25T19:49:39.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.345+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6a70001930 con 0x7f6a800b1730 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.345+0000 7f6a67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.345+0000 7f6a67fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.345+0000 7f6a67fff640 10 monclient(hunting): dump: 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:39.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.345+0000 7f6a67fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.354+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2161408993 0 0) 0x7f6a6c004eb0 con 0x7f6a800b4d60 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.354+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a80136af0 con 0x7f6a800b4d60 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.354+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6a74001b00 con 0x7f6a800b4d60 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.354+0000 7f6a67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.354+0000 7f6a67fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.354+0000 7f6a67fff640 10 monclient(hunting): dump: 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:39.354 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:39.355 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:39.355 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:39.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.355+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 866398894 0 0) 0x7f6a80136250 con 0x7f6a8014be90 2024-03-25T19:49:39.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.355+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a6c004eb0 con 0x7f6a8014be90 2024-03-25T19:49:39.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.355+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6a78001c10 con 0x7f6a8014be90 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.355+0000 7f6a67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.355+0000 7f6a67fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.355+0000 7f6a67fff640 10 monclient(hunting): dump: 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:39.357 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.359+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2088628708 0 0) 0x7f6a80136c80 con 0x7f6a800b1730 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.359+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 >> v1:172.21.15.53:6789/0 conn(0x7f6a8014be90 legacy=0x7f6a801e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.359+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 >> v1:172.21.15.18:6789/0 conn(0x7f6a800b4d60 legacy=0x7f6a8012fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.359+0000 7f6a67fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.359+0000 7f6a67fff640 1 monclient: found mon.c 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a80138210 con 0x7f6a800b1730 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a67fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a67fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 10 monclient: _renew_subs 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a801384c0 con 0x7f6a800b1730 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 10 monclient: _renew_subs 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a80145700 con 0x7f6a800b1730 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.360+0000 7f6a85d4d640 1 librados: init done 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.361+0000 7f6a85d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.361+0000 7f6a85d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.361+0000 7f6a85d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:39.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.361+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a801360a0 con 0x7f6a800b1730 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.665+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a700047e0 con 0x7f6a800b1730 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.665+0000 7f6a67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.665+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6a70001d80 con 0x7f6a800b1730 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.665+0000 7f6a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.665+0000 7f6a67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.665+0000 7f6a67fff640 10 monclient: dump: 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:39.665 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.665+0000 7f6a7dffb640 4 set_mon_vals no callback set 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.665+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f6a7001a9e0 con 0x7f6a800b1730 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.666+0000 7f6a7effd640 10 monclient: get_auth_request con 0x7f6a6c06e060 auth_method 0 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.666+0000 7f6a67fff640 10 client.4589.objecter ms_handle_connect 0x7f6a6c06e060 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.672+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f6a700546d0 con 0x7f6a800b1730 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.672+0000 7f6a67fff640 10 client.4589.objecter ms_dispatch 0x7f6a80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.672+0000 7f6a67fff640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.672+0000 7f6a67fff640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.672+0000 7f6a67fff640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.672+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a70050010 con 0x7f6a800b1730 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.672+0000 7f6a67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.672+0000 7f6a67fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:39.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.895+0000 7f6a85d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:39.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.895+0000 7f6a85d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:39.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.895+0000 7f6a85d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:49:39.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.895+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a80135ec0 con 0x7f6a800b1730 2024-03-25T19:49:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.938+0000 7f6a67fff640 1 -- 172.21.15.18:0/1802180403 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f6a70052110 con 0x7f6a800b1730 2024-03-25T19:49:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.938+0000 7f6a67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.938+0000 7f6a67fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:39.939 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:49:39.939 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.940+0000 7f6a85d4d640 10 librados: watch_flush enter 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.940+0000 7f6a85d4d640 10 librados: watch_flush exit 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.940+0000 7f6a85d4d640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.940+0000 7f6a85d4d640 10 client.4589.objecter successfully canceled tick 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 >> v1:172.21.15.18:6800/1520000573 conn(0x7f6a6c06e060 legacy=0x7f6a6c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 10 monclient: shutdown 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 >> v1:172.21.15.18:6790/0 conn(0x7f6a800b1730 legacy=0x7f6a801d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 shutdown_connections 2024-03-25T19:49:39.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 >> 172.21.15.18:0/1802180403 conn(0x7f6a800617a0 msgr2=0x7f6a800b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:39.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 shutdown_connections 2024-03-25T19:49:39.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 1 -- 172.21.15.18:0/1802180403 wait complete. 2024-03-25T19:49:39.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:39.941+0000 7f6a85d4d640 1 librados: shutdown 2024-03-25T19:49:39.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:46.761 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.869+0000 7f75c3c86640 10 monclient: get_monmap_and_config 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.869+0000 7f75c3c86640 10 monclient: build_initial_monmap 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.869+0000 7f75c3c86640 10 monclient: monmap: 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:49:46.870402+0000 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:49:46.870402+0000 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:46.869 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:46.870 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:49:46.870 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:49:46.870 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 1 Processor -- start 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 1 -- start start 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 10 monclient: init 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 10 monclient(hunting): picked mon.noname-b con 0x7f75bc0b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 10 monclient(hunting): picked mon.noname-c con 0x7f75bc14be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0af940 con 0x7f75bc0b1730 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0b0a70 con 0x7f75bc14be90 2024-03-25T19:49:46.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc0b0d50 con 0x7f75bc0b4d60 2024-03-25T19:49:46.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:46.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.870+0000 7f75c3c86640 10 monclient(hunting): authenticate will time out at 5413.997727s 2024-03-25T19:49:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.904+0000 7f75c2223640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f75bc14be90 0x7f75bc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:48958/0 (socket says 172.21.15.18:48958) 2024-03-25T19:49:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.904+0000 7f75c2223640 1 -- 172.21.15.18:0/2815633704 learned_addr learned my addr 172.21.15.18:0/2815633704 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:46.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999618819 0 0) 0x7f75bc0b0d50 con 0x7f75bc0b4d60 2024-03-25T19:49:46.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 10 monclient(hunting): _negotiate 2024-03-25T19:49:46.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:49:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a8003520 con 0x7f75bc0b4d60 2024-03-25T19:49:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494757985 0 0) 0x7f75bc0b0a70 con 0x7f75bc14be90 2024-03-25T19:49:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 10 monclient(hunting): _negotiate 2024-03-25T19:49:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:46.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:49:46.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.921+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc0b0d50 con 0x7f75bc14be90 2024-03-25T19:49:46.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2546880622 0 0) 0x7f75bc0b0d50 con 0x7f75bc14be90 2024-03-25T19:49:46.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 >> v1:172.21.15.18:6790/0 conn(0x7f75bc0b4d60 legacy=0x7f75bc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:46.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 >> v1:172.21.15.18:6789/0 conn(0x7f75bc0b1730 legacy=0x7f75bc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 10 monclient: _finish_hunting 0 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 1 monclient: found mon.noname-c 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0b1030 con 0x7f75bc14be90 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 10 monclient: _finish_auth 0 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 10 monclient: _check_auth_tickets 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f75b40019f0 con 0x7f75bc14be90 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c3c86640 5 monclient: authenticate success, global_id 4594 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c3c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c0a20640 10 monclient: dump: 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:46.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.929+0000 7f75c3c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:46.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.944+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75b4001b50 con 0x7f75bc14be90 2024-03-25T19:49:46.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.944+0000 7f75c0a20640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:46.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.944+0000 7f75c0a20640 1 -- 172.21.15.18:0/2815633704 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f75b4001fa0 con 0x7f75bc14be90 2024-03-25T19:49:46.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.944+0000 7f75c3c86640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.944+0000 7f75c3c86640 4 set_mon_vals no callback set 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.944+0000 7f75c0a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.944+0000 7f75c0a20640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.944+0000 7f75c0a20640 10 monclient: dump: 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:46.945 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.945+0000 7f75c3c86640 10 monclient: shutdown 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.945+0000 7f75c3c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.945+0000 7f75c3c86640 1 -- 172.21.15.18:0/2815633704 >> v1:172.21.15.53:6789/0 conn(0x7f75bc14be90 legacy=0x7f75bc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.945+0000 7f75c3c86640 1 -- 172.21.15.18:0/2815633704 shutdown_connections 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.946+0000 7f75c3c86640 1 -- 172.21.15.18:0/2815633704 >> 172.21.15.18:0/2815633704 conn(0x7f75bc0617a0 msgr2=0x7f75bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.946+0000 7f75c3c86640 1 -- 172.21.15.18:0/2815633704 shutdown_connections 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.946+0000 7f75c3c86640 1 -- 172.21.15.18:0/2815633704 wait complete. 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.946+0000 7f75c3c86640 10 monclient: build_initial_monmap 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.946+0000 7f75c3c86640 10 monclient: monmap: 2024-03-25T19:49:46.947 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.947+0000 7f75c3c86640 1 librados: starting msgr at 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.947+0000 7f75c3c86640 1 librados: starting objecter 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.947+0000 7f75c3c86640 1 Processor -- start 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.947+0000 7f75c3c86640 1 -- start start 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.947+0000 7f75c3c86640 1 librados: setting wanted keys 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.947+0000 7f75c3c86640 1 librados: calling monclient init 2024-03-25T19:49:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.947+0000 7f75c3c86640 10 monclient: init 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 10 monclient(hunting): picked mon.noname-c con 0x7f75bc0b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 10 monclient(hunting): picked mon.noname-b con 0x7f75bc14be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc141ca0 con 0x7f75bc0b1730 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc143420 con 0x7f75bc14be90 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc140810 con 0x7f75bc0b4d60 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.948+0000 7f75c3c86640 10 monclient(hunting): authenticate will time out at 5414.075031s 2024-03-25T19:49:46.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.959+0000 7f75c2223640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f75bc14be90 0x7f75bc1d7900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:48968/0 (socket says 172.21.15.18:48968) 2024-03-25T19:49:46.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.959+0000 7f75c2223640 1 -- 172.21.15.18:0/3302962132 learned_addr learned my addr 172.21.15.18:0/3302962132 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:46.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.960+0000 7f75b27fc640 10 client.?.objecter ms_handle_connect 0x7f75bc14be90 2024-03-25T19:49:46.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.960+0000 7f75b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:46.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.961+0000 7f75b27fc640 10 client.?.objecter ms_handle_connect 0x7f75bc0b4d60 2024-03-25T19:49:46.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.961+0000 7f75b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.964+0000 7f75b27fc640 10 client.?.objecter ms_handle_connect 0x7f75bc0b1730 2024-03-25T19:49:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.964+0000 7f75b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.968+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741471476 0 0) 0x7f75bc143420 con 0x7f75bc14be90 2024-03-25T19:49:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.968+0000 7f75b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.968+0000 7f75b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.968+0000 7f75b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.968+0000 7f75b27fc640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.968+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a0004eb0 con 0x7f75bc14be90 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522463520 0 0) 0x7f75bc140810 con 0x7f75bc0b4d60 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc143420 con 0x7f75bc0b4d60 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383869116 0 0) 0x7f75bc141ca0 con 0x7f75bc0b1730 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.972+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc140810 con 0x7f75bc0b1730 2024-03-25T19:49:46.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.979+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 803655211 0 0) 0x7f75a0004eb0 con 0x7f75bc14be90 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.979+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75bc141ca0 con 0x7f75bc14be90 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.979+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f75b4001f20 con 0x7f75bc14be90 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.979+0000 7f75b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.979+0000 7f75b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.979+0000 7f75b27fc640 10 monclient(hunting): dump: 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:46.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.979+0000 7f75b27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.981+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2105369683 0 0) 0x7f75bc140810 con 0x7f75bc0b1730 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.981+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75a0004eb0 con 0x7f75bc0b1730 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.981+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f75b8001950 con 0x7f75bc0b1730 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.981+0000 7f75b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.981+0000 7f75b27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.981+0000 7f75b27fc640 10 monclient(hunting): dump: 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:46.981 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:46.982 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:46.982 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:46.982 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:46.982 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:46.982 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:46.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 773204292 0 0) 0x7f75bc141ca0 con 0x7f75bc14be90 2024-03-25T19:49:46.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 >> v1:172.21.15.18:6790/0 conn(0x7f75bc0b4d60 legacy=0x7f75bc132e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 >> v1:172.21.15.18:6789/0 conn(0x7f75bc0b1730 legacy=0x7f75bc1412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 1 monclient: found mon.b 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc140380 con 0x7f75bc14be90 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 10 monclient: _renew_subs 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:46.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc1405d0 con 0x7f75bc14be90 2024-03-25T19:49:46.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 10 client.4597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:46.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 10 monclient: _renew_subs 2024-03-25T19:49:46.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:46.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc133710 con 0x7f75bc14be90 2024-03-25T19:49:46.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.985+0000 7f75c3c86640 1 librados: init done 2024-03-25T19:49:46.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.986+0000 7f75c3c86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:46.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.986+0000 7f75c3c86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:46.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.986+0000 7f75c3c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:46.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:46.986+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75bc141a00 con 0x7f75bc14be90 2024-03-25T19:49:47.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.287+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75b4004e10 con 0x7f75bc14be90 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.287+0000 7f75b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.287+0000 7f75b3fff640 4 set_mon_vals no callback set 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.288+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f75b4004f80 con 0x7f75bc14be90 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.288+0000 7f75b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.288+0000 7f75b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.288+0000 7f75b27fc640 10 monclient: dump: 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:47.288 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:47.289 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:47.289 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:47.289 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:47.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.288+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f75b401af50 con 0x7f75bc14be90 2024-03-25T19:49:47.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.289+0000 7f75c1a22640 10 monclient: get_auth_request con 0x7f75a006df20 auth_method 0 2024-03-25T19:49:47.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.289+0000 7f75b27fc640 10 client.4597.objecter ms_handle_connect 0x7f75a006df20 2024-03-25T19:49:47.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.296+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f75b4054a50 con 0x7f75bc14be90 2024-03-25T19:49:47.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.296+0000 7f75b27fc640 10 client.4597.objecter ms_dispatch 0x7f75bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:47.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.296+0000 7f75b27fc640 3 client.4597.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:47.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.296+0000 7f75b27fc640 3 client.4597.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:47.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.296+0000 7f75b27fc640 20 client.4597.objecter dump_active .. 0 homeless 2024-03-25T19:49:47.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.297+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75b4050560 con 0x7f75bc14be90 2024-03-25T19:49:47.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.297+0000 7f75b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.297+0000 7f75b27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:47.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.545+0000 7f75c3c86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.545+0000 7f75c3c86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.545+0000 7f75c3c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:47.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.545+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75bc1d7620 con 0x7f75bc14be90 2024-03-25T19:49:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.570+0000 7f75b27fc640 1 -- 172.21.15.18:0/3302962132 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f75bc1d7620 con 0x7f75bc14be90 2024-03-25T19:49:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.570+0000 7f75b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.570+0000 7f75b27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:47.570 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:49:47.571 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:47.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.572+0000 7f75c3c86640 10 librados: watch_flush enter 2024-03-25T19:49:47.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 10 librados: watch_flush exit 2024-03-25T19:49:47.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 20 client.4597.objecter shutdown clearing up homeless session... 2024-03-25T19:49:47.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 10 client.4597.objecter successfully canceled tick 2024-03-25T19:49:47.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 >> v1:172.21.15.18:6800/1520000573 conn(0x7f75a006df20 legacy=0x7f75a00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 10 monclient: shutdown 2024-03-25T19:49:47.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:47.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 >> v1:172.21.15.53:6789/0 conn(0x7f75bc14be90 legacy=0x7f75bc1d7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 shutdown_connections 2024-03-25T19:49:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 >> 172.21.15.18:0/3302962132 conn(0x7f75bc0617a0 msgr2=0x7f75bc0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 shutdown_connections 2024-03-25T19:49:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 1 -- 172.21.15.18:0/3302962132 wait complete. 2024-03-25T19:49:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:47.573+0000 7f75c3c86640 1 librados: shutdown 2024-03-25T19:49:47.584 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:53.707 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:21.237 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.712 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:21.243 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.749 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:21.280 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.762 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:21.259 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.771 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:21.301 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.775 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:21.272 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.776 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:21.306 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.798 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:21.294 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:53.800 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:21.331 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:54.391 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:54.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.509+0000 7f23e7aa8640 10 monclient: get_monmap_and_config 2024-03-25T19:49:54.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.509+0000 7f23e7aa8640 10 monclient: build_initial_monmap 2024-03-25T19:49:54.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.509+0000 7f23e7aa8640 10 monclient: monmap: 2024-03-25T19:49:54.510 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:54.510 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:54.510 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:49:54.511358+0000 2024-03-25T19:49:54.510 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:49:54.511358+0000 2024-03-25T19:49:54.510 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:54.511 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:54.511 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:54.511 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:49:54.511 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:49:54.511 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:54.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.510+0000 7f23e7aa8640 1 Processor -- start 2024-03-25T19:49:54.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.510+0000 7f23e7aa8640 1 -- start start 2024-03-25T19:49:54.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.510+0000 7f23e7aa8640 10 monclient: init 2024-03-25T19:49:54.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:54.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:54.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f23e00a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:54.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f23e013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:54.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f23e01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:54.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e0142520 con 0x7f23e01430c0 2024-03-25T19:49:54.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e00b28a0 con 0x7f23e013f710 2024-03-25T19:49:54.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e00b2a30 con 0x7f23e00a98f0 2024-03-25T19:49:54.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:54.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.511+0000 7f23e7aa8640 10 monclient(hunting): authenticate will time out at 5421.638966s 2024-03-25T19:49:54.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.517+0000 7f23e5043640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f23e013f710 0x7f23e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:53498/0 (socket says 172.21.15.18:53498) 2024-03-25T19:49:54.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.517+0000 7f23e5043640 1 -- 172.21.15.18:0/108486264 learned_addr learned my addr 172.21.15.18:0/108486264 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:54.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.531+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247741984 0 0) 0x7f23e00b28a0 con 0x7f23e013f710 2024-03-25T19:49:54.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.531+0000 7f23e4842640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.531+0000 7f23e4842640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.531+0000 7f23e4842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.531+0000 7f23e4842640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:49:54.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.531+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc003520 con 0x7f23e013f710 2024-03-25T19:49:54.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.533+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986974273 0 0) 0x7f23e0142520 con 0x7f23e01430c0 2024-03-25T19:49:54.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.533+0000 7f23e4842640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.533+0000 7f23e4842640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.533+0000 7f23e4842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.533+0000 7f23e4842640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:49:54.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.533+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e00b28a0 con 0x7f23e01430c0 2024-03-25T19:49:54.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.537+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730903555 0 0) 0x7f23e00b2a30 con 0x7f23e00a98f0 2024-03-25T19:49:54.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.537+0000 7f23e4842640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.537+0000 7f23e4842640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.537+0000 7f23e4842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.537+0000 7f23e4842640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:49:54.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.537+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e0142520 con 0x7f23e00a98f0 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1328504678 0 0) 0x7f23cc003520 con 0x7f23e013f710 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 >> v1:172.21.15.18:6790/0 conn(0x7f23e00a98f0 legacy=0x7f23e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 >> v1:172.21.15.18:6789/0 conn(0x7f23e01430c0 legacy=0x7f23e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 10 monclient: _finish_hunting 0 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 1 monclient: found mon.noname-c 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e00b44c0 con 0x7f23e013f710 2024-03-25T19:49:54.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 10 monclient: _finish_auth 0 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 10 monclient: _check_auth_tickets 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f23d0001690 con 0x7f23e013f710 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e7aa8640 5 monclient: authenticate success, global_id 4600 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e7aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e4842640 10 monclient: dump: 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:54.544 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:54.545 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:54.545 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:54.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.542+0000 7f23e7aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:54.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.548+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23d00017f0 con 0x7f23e013f710 2024-03-25T19:49:54.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.548+0000 7f23e4842640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.548+0000 7f23e4842640 1 -- 172.21.15.18:0/108486264 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f23d0001c00 con 0x7f23e013f710 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.548+0000 7f23e7aa8640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.548+0000 7f23e7aa8640 4 set_mon_vals no callback set 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.548+0000 7f23e4842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.548+0000 7f23e4842640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.548+0000 7f23e4842640 10 monclient: dump: 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:54.550 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:54.551 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:54.551 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:54.551 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:54.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.549+0000 7f23e7aa8640 10 monclient: shutdown 2024-03-25T19:49:54.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.549+0000 7f23e7aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.549+0000 7f23e7aa8640 1 -- 172.21.15.18:0/108486264 >> v1:172.21.15.53:6789/0 conn(0x7f23e013f710 legacy=0x7f23e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.549+0000 7f23e7aa8640 1 -- 172.21.15.18:0/108486264 shutdown_connections 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.549+0000 7f23e7aa8640 1 -- 172.21.15.18:0/108486264 >> 172.21.15.18:0/108486264 conn(0x7f23e00b7ea0 msgr2=0x7f23e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.550+0000 7f23e7aa8640 1 -- 172.21.15.18:0/108486264 shutdown_connections 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.550+0000 7f23e7aa8640 1 -- 172.21.15.18:0/108486264 wait complete. 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.550+0000 7f23e7aa8640 10 monclient: build_initial_monmap 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.550+0000 7f23e7aa8640 10 monclient: monmap: 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:49:54.553 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.551+0000 7f23e7aa8640 1 librados: starting msgr at 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.551+0000 7f23e7aa8640 1 librados: starting objecter 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.551+0000 7f23e7aa8640 1 Processor -- start 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.551+0000 7f23e7aa8640 1 -- start start 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.551+0000 7f23e7aa8640 1 librados: setting wanted keys 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.551+0000 7f23e7aa8640 1 librados: calling monclient init 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.551+0000 7f23e7aa8640 10 monclient: init 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f23e00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f23e013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f23e01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e0139a40 con 0x7f23e00a98f0 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e0133f20 con 0x7f23e013f710 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e00ab950 con 0x7f23e01430c0 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:54.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.552+0000 7f23e7aa8640 10 monclient(hunting): authenticate will time out at 5421.679903s 2024-03-25T19:49:54.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.556+0000 7f23e5043640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f23e013f710 0x7f23e0133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:53532/0 (socket says 172.21.15.18:53532) 2024-03-25T19:49:54.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.556+0000 7f23e5043640 1 -- 172.21.15.18:0/2668190840 learned_addr learned my addr 172.21.15.18:0/2668190840 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:49:54.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.564+0000 7f23d67fc640 10 client.?.objecter ms_handle_connect 0x7f23e013f710 2024-03-25T19:49:54.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.564+0000 7f23d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.567+0000 7f23d67fc640 10 client.?.objecter ms_handle_connect 0x7f23e00a98f0 2024-03-25T19:49:54.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.567+0000 7f23d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.574+0000 7f23d67fc640 10 client.?.objecter ms_handle_connect 0x7f23e01430c0 2024-03-25T19:49:54.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.574+0000 7f23d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.577+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423727711 0 0) 0x7f23e0133f20 con 0x7f23e013f710 2024-03-25T19:49:54.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.577+0000 7f23d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.577+0000 7f23d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.577+0000 7f23d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.577+0000 7f23d67fc640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:49:54.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.577+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4004eb0 con 0x7f23e013f710 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3362784738 0 0) 0x7f23c4004eb0 con 0x7f23e013f710 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23e0133f20 con 0x7f23e013f710 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f23d0001bc0 con 0x7f23e013f710 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 10 monclient(hunting): dump: 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:54.591 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352057565 0 0) 0x7f23e00ab950 con 0x7f23e01430c0 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:49:54.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.589+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4004eb0 con 0x7f23e01430c0 2024-03-25T19:49:54.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1239253833 0 0) 0x7f23e0133f20 con 0x7f23e013f710 2024-03-25T19:49:54.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 >> v1:172.21.15.18:6790/0 conn(0x7f23e01430c0 legacy=0x7f23e0133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 >> v1:172.21.15.18:6789/0 conn(0x7f23e00a98f0 legacy=0x7f23e0138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 1 monclient: found mon.b 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e00abc30 con 0x7f23e013f710 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 10 monclient: _renew_subs 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e00abf10 con 0x7f23e013f710 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:54.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 10 monclient: _renew_subs 2024-03-25T19:49:54.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:54.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e0139370 con 0x7f23e013f710 2024-03-25T19:49:54.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.602+0000 7f23e7aa8640 1 librados: init done 2024-03-25T19:49:54.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.604+0000 7f23e7aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:54.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.604+0000 7f23e7aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:54.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.604+0000 7f23e7aa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:54.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.604+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e00627b0 con 0x7f23e013f710 2024-03-25T19:49:54.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.901+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23d00011f0 con 0x7f23e013f710 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.901+0000 7f23d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.901+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f23d0004c70 con 0x7f23e013f710 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.901+0000 7f23d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.901+0000 7f23d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.901+0000 7f23d67fc640 10 monclient: dump: 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:54.902 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:49:54.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.901+0000 7f23d7fff640 4 set_mon_vals no callback set 2024-03-25T19:49:54.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.902+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f23d0004e90 con 0x7f23e013f710 2024-03-25T19:49:54.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.902+0000 7f23e5844640 10 monclient: get_auth_request con 0x7f23c406dd40 auth_method 0 2024-03-25T19:49:54.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.902+0000 7f23d67fc640 10 client.4603.objecter ms_handle_connect 0x7f23c406dd40 2024-03-25T19:49:54.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.907+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f23d0054910 con 0x7f23e013f710 2024-03-25T19:49:54.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.907+0000 7f23d67fc640 10 client.4603.objecter ms_dispatch 0x7f23e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:54.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.907+0000 7f23d67fc640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:54.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.907+0000 7f23d67fc640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:54.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.907+0000 7f23d67fc640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T19:49:54.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.907+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23d0050250 con 0x7f23e013f710 2024-03-25T19:49:54.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.907+0000 7f23d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:54.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:54.907+0000 7f23d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:55.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.135+0000 7f23e7aa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.135+0000 7f23e7aa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.135+0000 7f23e7aa8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:49:55.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.135+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23e00b7180 con 0x7f23e013f710 2024-03-25T19:49:55.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.194+0000 7f23d67fc640 1 -- 172.21.15.18:0/2668190840 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f23d0052350 con 0x7f23e013f710 2024-03-25T19:49:55.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.194+0000 7f23d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.194+0000 7f23d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:55.195 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:49:55.195 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:55.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 10 librados: watch_flush enter 2024-03-25T19:49:55.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 10 librados: watch_flush exit 2024-03-25T19:49:55.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T19:49:55.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 10 client.4603.objecter successfully canceled tick 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 >> v1:172.21.15.18:6800/1520000573 conn(0x7f23c406dd40 legacy=0x7f23c4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 10 monclient: shutdown 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 >> v1:172.21.15.53:6789/0 conn(0x7f23e013f710 legacy=0x7f23e0133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 shutdown_connections 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 >> 172.21.15.18:0/2668190840 conn(0x7f23e00b7ea0 msgr2=0x7f23e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 shutdown_connections 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 1 -- 172.21.15.18:0/2668190840 wait complete. 2024-03-25T19:49:55.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:49:55.196+0000 7f23e7aa8640 1 librados: shutdown 2024-03-25T19:49:55.208 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:02.013 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:02.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.121+0000 7f171d2e2640 10 monclient: get_monmap_and_config 2024-03-25T19:50:02.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.121+0000 7f171d2e2640 10 monclient: build_initial_monmap 2024-03-25T19:50:02.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.121+0000 7f171d2e2640 10 monclient: monmap: 2024-03-25T19:50:02.122 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:02.122 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:02.122 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:50:02.123210+0000 2024-03-25T19:50:02.122 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:50:02.123210+0000 2024-03-25T19:50:02.122 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:02.123 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:02.123 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:02.123 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:50:02.123 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:50:02.123 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:02.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.121+0000 7f171d2e2640 1 Processor -- start 2024-03-25T19:50:02.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 1 -- start start 2024-03-25T19:50:02.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 10 monclient: init 2024-03-25T19:50:02.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:02.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:02.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 10 monclient(hunting): picked mon.noname-c con 0x7f17180b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:02.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 10 monclient(hunting): picked mon.noname-b con 0x7f17180b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:02.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 10 monclient(hunting): picked mon.noname-a con 0x7f171814be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:02.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17180af940 con 0x7f171814be90 2024-03-25T19:50:02.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17180b0a70 con 0x7f17180b4d60 2024-03-25T19:50:02.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17180b0d50 con 0x7f17180b1730 2024-03-25T19:50:02.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:02.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.122+0000 7f171d2e2640 10 monclient(hunting): authenticate will time out at 5429.250350s 2024-03-25T19:50:02.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.123+0000 7f1716ffd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f17180b4d60 0x7f171813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:46216/0 (socket says 172.21.15.18:46216) 2024-03-25T19:50:02.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.123+0000 7f1716ffd640 1 -- 172.21.15.18:0/3886821486 learned_addr learned my addr 172.21.15.18:0/3886821486 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:02.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.187+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182477045 0 0) 0x7f17180af940 con 0x7f171814be90 2024-03-25T19:50:02.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.188+0000 7f1715ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.188+0000 7f1715ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.188+0000 7f1715ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.188+0000 7f1715ffb640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:50:02.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.188+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1700003520 con 0x7f171814be90 2024-03-25T19:50:02.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3198459777 0 0) 0x7f1700003520 con 0x7f171814be90 2024-03-25T19:50:02.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 >> v1:172.21.15.18:6790/0 conn(0x7f17180b1730 legacy=0x7f17180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 >> v1:172.21.15.53:6789/0 conn(0x7f17180b4d60 legacy=0x7f171813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 1 monclient: found mon.noname-a 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17180b1030 con 0x7f171814be90 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1710001a10 con 0x7f171814be90 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f1715ffb640 10 monclient: dump: 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:02.199 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f171d2e2640 5 monclient: authenticate success, global_id 4632 2024-03-25T19:50:02.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.197+0000 7f171d2e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.235+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1710001b70 con 0x7f171814be90 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.235+0000 7f1715ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.235+0000 7f1715ffb640 1 -- 172.21.15.18:0/3886821486 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1710001f80 con 0x7f171814be90 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.235+0000 7f171d2e2640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.235+0000 7f171d2e2640 4 set_mon_vals no callback set 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.235+0000 7f1715ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.235+0000 7f1715ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.235+0000 7f1715ffb640 10 monclient: dump: 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:02.236 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.237 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.237 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.237 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:02.237 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:02.237 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:02.237 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:02.237 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.236+0000 7f171d2e2640 10 monclient: shutdown 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.236+0000 7f171d2e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.236+0000 7f171d2e2640 1 -- 172.21.15.18:0/3886821486 >> v1:172.21.15.18:6789/0 conn(0x7f171814be90 legacy=0x7f171814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.236+0000 7f171d2e2640 1 -- 172.21.15.18:0/3886821486 shutdown_connections 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.236+0000 7f171d2e2640 1 -- 172.21.15.18:0/3886821486 >> 172.21.15.18:0/3886821486 conn(0x7f17180617a0 msgr2=0x7f1718061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.236+0000 7f171d2e2640 1 -- 172.21.15.18:0/3886821486 shutdown_connections 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.236+0000 7f171d2e2640 1 -- 172.21.15.18:0/3886821486 wait complete. 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.237+0000 7f171d2e2640 10 monclient: build_initial_monmap 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.237+0000 7f171d2e2640 10 monclient: monmap: 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:50:02.239 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.237+0000 7f171d2e2640 1 librados: starting msgr at 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.237+0000 7f171d2e2640 1 librados: starting objecter 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 1 Processor -- start 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 1 -- start start 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 1 librados: setting wanted keys 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 1 librados: calling monclient init 2024-03-25T19:50:02.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 10 monclient: init 2024-03-25T19:50:02.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:02.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:02.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 10 monclient(hunting): picked mon.noname-a con 0x7f17180b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:02.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.238+0000 7f171d2e2640 10 monclient(hunting): picked mon.noname-b con 0x7f17180b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:02.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.239+0000 7f171d2e2640 10 monclient(hunting): picked mon.noname-c con 0x7f171814be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:02.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.239+0000 7f171d2e2640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17181378a0 con 0x7f17180b1730 2024-03-25T19:50:02.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.239+0000 7f171d2e2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17181381a0 con 0x7f17180b4d60 2024-03-25T19:50:02.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.239+0000 7f171d2e2640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1718138480 con 0x7f171814be90 2024-03-25T19:50:02.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.239+0000 7f171d2e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:02.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.239+0000 7f171d2e2640 10 monclient(hunting): authenticate will time out at 5429.366916s 2024-03-25T19:50:02.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.248+0000 7f17167fc640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f17180b4d60 0x7f1718142b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:46262/0 (socket says 172.21.15.18:46262) 2024-03-25T19:50:02.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.248+0000 7f17167fc640 1 -- 172.21.15.18:0/2049495212 learned_addr learned my addr 172.21.15.18:0/2049495212 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:02.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.249+0000 7f16ff7fe640 10 client.?.objecter ms_handle_connect 0x7f17180b4d60 2024-03-25T19:50:02.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.249+0000 7f16ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.255+0000 7f16ff7fe640 10 client.?.objecter ms_handle_connect 0x7f17180b1730 2024-03-25T19:50:02.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.255+0000 7f16ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.265+0000 7f16ff7fe640 10 client.?.objecter ms_handle_connect 0x7f171814be90 2024-03-25T19:50:02.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.265+0000 7f16ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.332+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411602171 0 0) 0x7f17181378a0 con 0x7f17180b1730 2024-03-25T19:50:02.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.332+0000 7f16ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.332+0000 7f16ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.332+0000 7f16ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.332+0000 7f16ff7fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:50:02.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.332+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1704004eb0 con 0x7f17180b1730 2024-03-25T19:50:02.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.340+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814861171 0 0) 0x7f17181381a0 con 0x7f17180b4d60 2024-03-25T19:50:02.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.340+0000 7f16ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.340+0000 7f16ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.340+0000 7f16ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.340+0000 7f16ff7fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:50:02.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.340+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17181378a0 con 0x7f17180b4d60 2024-03-25T19:50:02.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.347+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4178950220 0 0) 0x7f1704004eb0 con 0x7f17180b1730 2024-03-25T19:50:02.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.347+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17181381a0 con 0x7f17180b1730 2024-03-25T19:50:02.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.347+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f17080018a0 con 0x7f17180b1730 2024-03-25T19:50:02.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.347+0000 7f16ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.347+0000 7f16ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:02.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.347+0000 7f16ff7fe640 10 monclient(hunting): dump: 2024-03-25T19:50:02.348 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:02.348 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.347+0000 7f16ff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:02.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.353+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209916703 0 0) 0x7f1718138480 con 0x7f171814be90 2024-03-25T19:50:02.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.353+0000 7f16ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.353+0000 7f16ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.353+0000 7f16ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.353+0000 7f16ff7fe640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:50:02.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.353+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1704004eb0 con 0x7f171814be90 2024-03-25T19:50:02.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.357+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4133163585 0 0) 0x7f17181378a0 con 0x7f17180b4d60 2024-03-25T19:50:02.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.357+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1718138480 con 0x7f17180b4d60 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.357+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f170c001970 con 0x7f17180b4d60 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.357+0000 7f16ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.357+0000 7f16ff7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.357+0000 7f16ff7fe640 10 monclient(hunting): dump: 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:02.359 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1372690473 0 0) 0x7f1718138480 con 0x7f17180b4d60 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 >> v1:172.21.15.18:6790/0 conn(0x7f171814be90 legacy=0x7f1718144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 >> v1:172.21.15.18:6789/0 conn(0x7f17180b1730 legacy=0x7f1718137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 1 monclient: found mon.b 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718138760 con 0x7f17180b4d60 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:02.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f16ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 10 monclient: _renew_subs 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1718143c40 con 0x7f17180b4d60 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 10 monclient: _renew_subs 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1718143410 con 0x7f17180b4d60 2024-03-25T19:50:02.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.370+0000 7f171d2e2640 1 librados: init done 2024-03-25T19:50:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.371+0000 7f171d2e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.371+0000 7f171d2e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.371+0000 7f171d2e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.371+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17180627b0 con 0x7f17180b4d60 2024-03-25T19:50:02.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.661+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f170c0035f0 con 0x7f17180b4d60 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.661+0000 7f16ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.661+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f170c003980 con 0x7f17180b4d60 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.661+0000 7f16ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.661+0000 7f16ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.661+0000 7f16ff7fe640 10 monclient: dump: 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:02.663 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:02.664 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:02.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.661+0000 7f17157fa640 4 set_mon_vals no callback set 2024-03-25T19:50:02.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.662+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f170c003ae0 con 0x7f17180b4d60 2024-03-25T19:50:02.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.663+0000 7f1716ffd640 10 monclient: get_auth_request con 0x7f170406df20 auth_method 0 2024-03-25T19:50:02.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.663+0000 7f16ff7fe640 10 client.4609.objecter ms_handle_connect 0x7f170406df20 2024-03-25T19:50:02.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.669+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f170c054810 con 0x7f17180b4d60 2024-03-25T19:50:02.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.669+0000 7f16ff7fe640 10 client.4609.objecter ms_dispatch 0x7f1718000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:02.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.669+0000 7f16ff7fe640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:02.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.669+0000 7f16ff7fe640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:02.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.669+0000 7f16ff7fe640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T19:50:02.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.681+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f170c050320 con 0x7f17180b4d60 2024-03-25T19:50:02.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.681+0000 7f16ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.681+0000 7f16ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:02.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.949+0000 7f171d2e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:02.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.949+0000 7f171d2e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:02.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.949+0000 7f171d2e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:02.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.949+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1718143dd0 con 0x7f17180b4d60 2024-03-25T19:50:02.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.990+0000 7f16ff7fe640 1 -- 172.21.15.18:0/2049495212 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f170c052250 con 0x7f17180b4d60 2024-03-25T19:50:02.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.990+0000 7f16ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:02.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.990+0000 7f16ff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:02.991 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:50:02.991 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:02.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.992+0000 7f171d2e2640 10 librados: watch_flush enter 2024-03-25T19:50:02.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.993+0000 7f171d2e2640 10 librados: watch_flush exit 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.993+0000 7f171d2e2640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.993+0000 7f171d2e2640 10 client.4609.objecter successfully canceled tick 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 >> v1:172.21.15.18:6800/1520000573 conn(0x7f170406df20 legacy=0x7f17040703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 10 monclient: shutdown 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 >> v1:172.21.15.53:6789/0 conn(0x7f17180b4d60 legacy=0x7f1718142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 shutdown_connections 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 >> 172.21.15.18:0/2049495212 conn(0x7f17180617a0 msgr2=0x7f17180b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 shutdown_connections 2024-03-25T19:50:02.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 1 -- 172.21.15.18:0/2049495212 wait complete. 2024-03-25T19:50:02.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:02.994+0000 7f171d2e2640 1 librados: shutdown 2024-03-25T19:50:03.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:09.817 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:09.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.931+0000 7fc2ce8d6640 10 monclient: get_monmap_and_config 2024-03-25T19:50:09.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.931+0000 7fc2ce8d6640 10 monclient: build_initial_monmap 2024-03-25T19:50:09.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.931+0000 7fc2ce8d6640 10 monclient: monmap: 2024-03-25T19:50:09.931 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:09.931 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:09.931 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:50:09.932430+0000 2024-03-25T19:50:09.931 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:50:09.932430+0000 2024-03-25T19:50:09.931 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:09.932 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:09.932 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:09.932 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:50:09.932 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:50:09.932 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:09.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.932+0000 7fc2ce8d6640 1 Processor -- start 2024-03-25T19:50:09.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.932+0000 7fc2ce8d6640 1 -- start start 2024-03-25T19:50:09.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.932+0000 7fc2ce8d6640 10 monclient: init 2024-03-25T19:50:09.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:09.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:09.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2c80a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:09.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc2c813f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:09.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc2c81430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:09.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c8142520 con 0x7fc2c813f710 2024-03-25T19:50:09.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c80b28a0 con 0x7fc2c80a98f0 2024-03-25T19:50:09.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c80b2a30 con 0x7fc2c81430c0 2024-03-25T19:50:09.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:09.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.933+0000 7fc2ce8d6640 10 monclient(hunting): authenticate will time out at 5437.060080s 2024-03-25T19:50:09.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.940+0000 7fc2cce73640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc2c81430c0 0x7fc2c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35618/0 (socket says 172.21.15.18:35618) 2024-03-25T19:50:09.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:09.940+0000 7fc2cce73640 1 -- 172.21.15.18:0/1483471199 learned_addr learned my addr 172.21.15.18:0/1483471199 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:10.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.023+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552051670 0 0) 0x7fc2c8142520 con 0x7fc2c813f710 2024-03-25T19:50:10.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.023+0000 7fc2c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.023+0000 7fc2c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.023+0000 7fc2c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.023+0000 7fc2c6ffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:50:10.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.023+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b0003520 con 0x7fc2c813f710 2024-03-25T19:50:10.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 214993706 0 0) 0x7fc2b0003520 con 0x7fc2c813f710 2024-03-25T19:50:10.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 >> v1:172.21.15.18:6790/0 conn(0x7fc2c81430c0 legacy=0x7fc2c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 >> v1:172.21.15.53:6789/0 conn(0x7fc2c80a98f0 legacy=0x7fc2c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c80b44c0 con 0x7fc2c813f710 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc2bc001690 con 0x7fc2c813f710 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2ce8d6640 5 monclient: authenticate success, global_id 4638 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2ce8d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2c6ffd640 10 monclient: dump: 2024-03-25T19:50:10.061 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:10.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.060+0000 7fc2ce8d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.083+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2bc0017f0 con 0x7fc2c813f710 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.083+0000 7fc2c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.083+0000 7fc2c6ffd640 1 -- 172.21.15.18:0/1483471199 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc2bc001c00 con 0x7fc2c813f710 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.083+0000 7fc2c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.083+0000 7fc2c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.083+0000 7fc2c6ffd640 10 monclient: dump: 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.083+0000 7fc2ce8d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.083+0000 7fc2ce8d6640 4 set_mon_vals no callback set 2024-03-25T19:50:10.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.084+0000 7fc2ce8d6640 10 monclient: shutdown 2024-03-25T19:50:10.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.084+0000 7fc2ce8d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:10.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.084+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/1483471199 >> v1:172.21.15.18:6789/0 conn(0x7fc2c813f710 legacy=0x7fc2c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.084+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/1483471199 shutdown_connections 2024-03-25T19:50:10.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.084+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/1483471199 >> 172.21.15.18:0/1483471199 conn(0x7fc2c80b7ea0 msgr2=0x7fc2c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.084+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/1483471199 shutdown_connections 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.084+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/1483471199 wait complete. 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.085+0000 7fc2ce8d6640 10 monclient: build_initial_monmap 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.085+0000 7fc2ce8d6640 10 monclient: monmap: 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:50:10.086 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.085+0000 7fc2ce8d6640 1 librados: starting msgr at 2024-03-25T19:50:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.085+0000 7fc2ce8d6640 1 librados: starting objecter 2024-03-25T19:50:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.085+0000 7fc2ce8d6640 1 Processor -- start 2024-03-25T19:50:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 1 -- start start 2024-03-25T19:50:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 1 librados: setting wanted keys 2024-03-25T19:50:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 1 librados: calling monclient init 2024-03-25T19:50:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 10 monclient: init 2024-03-25T19:50:10.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:10.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:10.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc2c80a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:10.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc2c813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:10.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc2c81430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:10.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c80abd70 con 0x7fc2c81430c0 2024-03-25T19:50:10.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c81d7710 con 0x7fc2c813f710 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c81d8340 con 0x7fc2c80a98f0 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.086+0000 7fc2ce8d6640 10 monclient(hunting): authenticate will time out at 5437.213601s 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.087+0000 7fc2c7fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc2c80a98f0 0x7fc2c8137be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35698/0 (socket says 172.21.15.18:35698) 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.087+0000 7fc2c7fff640 1 -- 172.21.15.18:0/3718140384 learned_addr learned my addr 172.21.15.18:0/3718140384 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.090+0000 7fc2c4ff9640 10 client.?.objecter ms_handle_connect 0x7fc2c80a98f0 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.090+0000 7fc2c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.096+0000 7fc2c4ff9640 10 client.?.objecter ms_handle_connect 0x7fc2c81430c0 2024-03-25T19:50:10.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.096+0000 7fc2c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.108+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633459334 0 0) 0x7fc2c80abd70 con 0x7fc2c81430c0 2024-03-25T19:50:10.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.108+0000 7fc2c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.108+0000 7fc2c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.108+0000 7fc2c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.108+0000 7fc2c4ff9640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:50:10.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.108+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b4004eb0 con 0x7fc2c81430c0 2024-03-25T19:50:10.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.122+0000 7fc2c4ff9640 10 client.?.objecter ms_handle_connect 0x7fc2c813f710 2024-03-25T19:50:10.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.122+0000 7fc2c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.177+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3519253454 0 0) 0x7fc2b4004eb0 con 0x7fc2c81430c0 2024-03-25T19:50:10.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.177+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2c80abd70 con 0x7fc2c81430c0 2024-03-25T19:50:10.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.177+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc2c0001b00 con 0x7fc2c81430c0 2024-03-25T19:50:10.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.177+0000 7fc2c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.177+0000 7fc2c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:10.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.177+0000 7fc2c4ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:10.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.177+0000 7fc2c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:10.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3100388454 0 0) 0x7fc2c80abd70 con 0x7fc2c81430c0 2024-03-25T19:50:10.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 >> v1:172.21.15.18:6790/0 conn(0x7fc2c80a98f0 legacy=0x7fc2c8137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 >> v1:172.21.15.53:6789/0 conn(0x7fc2c813f710 legacy=0x7fc2c81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:10.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 1 monclient: found mon.a 2024-03-25T19:50:10.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:10.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:10.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c81d84d0 con 0x7fc2c81430c0 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 5 monclient: authenticate success, global_id 4641 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 10 monclient: _renew_subs 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c81d41e0 con 0x7fc2c81430c0 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 10 client.4641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 10 monclient: _renew_subs 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c81d5150 con 0x7fc2c81430c0 2024-03-25T19:50:10.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.191+0000 7fc2ce8d6640 1 librados: init done 2024-03-25T19:50:10.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.192+0000 7fc2ce8d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.192+0000 7fc2ce8d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.192+0000 7fc2ce8d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:10.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.192+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c80627b0 con 0x7fc2c81430c0 2024-03-25T19:50:10.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.516+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2c0001fb0 con 0x7fc2c81430c0 2024-03-25T19:50:10.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.516+0000 7fc2c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.516+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fc2c0004d00 con 0x7fc2c81430c0 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.516+0000 7fc2c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.516+0000 7fc2c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.516+0000 7fc2c4ff9640 10 monclient: dump: 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.516+0000 7fc2c67fc640 4 set_mon_vals no callback set 2024-03-25T19:50:10.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.517+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fc2c0004f20 con 0x7fc2c81430c0 2024-03-25T19:50:10.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.518+0000 7fc2c7fff640 10 monclient: get_auth_request con 0x7fc2b406dc80 auth_method 0 2024-03-25T19:50:10.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.518+0000 7fc2c4ff9640 10 client.4641.objecter ms_handle_connect 0x7fc2b406dc80 2024-03-25T19:50:10.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.525+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fc2c00549e0 con 0x7fc2c81430c0 2024-03-25T19:50:10.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.525+0000 7fc2c4ff9640 10 client.4641.objecter ms_dispatch 0x7fc2c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:10.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.525+0000 7fc2c4ff9640 3 client.4641.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:10.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.525+0000 7fc2c4ff9640 3 client.4641.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:10.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.525+0000 7fc2c4ff9640 20 client.4641.objecter dump_active .. 0 homeless 2024-03-25T19:50:10.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.525+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2c00504f0 con 0x7fc2c81430c0 2024-03-25T19:50:10.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.525+0000 7fc2c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.525+0000 7fc2c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:10.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.781+0000 7fc2ce8d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.781+0000 7fc2ce8d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.781+0000 7fc2ce8d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:10.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.781+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2c81d7950 con 0x7fc2c81430c0 2024-03-25T19:50:10.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.816+0000 7fc2c4ff9640 1 -- 172.21.15.18:0/3718140384 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fc2c0052420 con 0x7fc2c81430c0 2024-03-25T19:50:10.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.816+0000 7fc2c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.816+0000 7fc2c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:10.816 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:50:10.816 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.818+0000 7fc2ce8d6640 10 librados: watch_flush enter 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.818+0000 7fc2ce8d6640 10 librados: watch_flush exit 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.818+0000 7fc2ce8d6640 20 client.4641.objecter shutdown clearing up homeless session... 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.818+0000 7fc2ce8d6640 10 client.4641.objecter successfully canceled tick 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 >> v1:172.21.15.18:6800/1520000573 conn(0x7fc2b406dc80 legacy=0x7fc2b4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 10 monclient: shutdown 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 >> v1:172.21.15.18:6789/0 conn(0x7fc2c81430c0 legacy=0x7fc2c81d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 shutdown_connections 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 >> 172.21.15.18:0/3718140384 conn(0x7fc2c80b7ea0 msgr2=0x7fc2c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:10.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 shutdown_connections 2024-03-25T19:50:10.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 1 -- 172.21.15.18:0/3718140384 wait complete. 2024-03-25T19:50:10.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:10.819+0000 7fc2ce8d6640 1 librados: shutdown 2024-03-25T19:50:10.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:13.768 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:41.299 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.776 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:41.307 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.818 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:41.349 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.831 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:41.327 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.836 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:41.333 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.842 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:41.373 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.846 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:41.377 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.865 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:09:41.361 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:13.882 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:41.412 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:17.638 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:17.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.755+0000 7f179a684640 10 monclient: get_monmap_and_config 2024-03-25T19:50:17.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.755+0000 7f179a684640 10 monclient: build_initial_monmap 2024-03-25T19:50:17.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.755+0000 7f179a684640 10 monclient: monmap: 2024-03-25T19:50:17.755 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:50:17.756568+0000 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:50:17.756568+0000 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:50:17.756 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:17.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 1 Processor -- start 2024-03-25T19:50:17.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 1 -- start start 2024-03-25T19:50:17.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 10 monclient: init 2024-03-25T19:50:17.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 10 monclient(hunting): picked mon.noname-c con 0x7f1794149f50 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 10 monclient(hunting): picked mon.noname-b con 0x7f179406c780 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 10 monclient(hunting): picked mon.noname-a con 0x7f17940b1b10 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179414a330 con 0x7f17940b1b10 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1794070660 con 0x7f1794149f50 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179406d7c0 con 0x7f179406c780 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.757+0000 7f179a684640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.758+0000 7f179a684640 10 monclient(hunting): authenticate will time out at 5444.884742s 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.759+0000 7f1793fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1794149f50 0x7f17940b0710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:59628/0 (socket says 172.21.15.18:59628) 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.759+0000 7f1793fff640 1 -- 172.21.15.18:0/2417700164 learned_addr learned my addr 172.21.15.18:0/2417700164 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.836+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116175226 0 0) 0x7f179414a330 con 0x7f17940b1b10 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.836+0000 7f17937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.836+0000 7f17937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.836+0000 7f17937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.836+0000 7f17937fe640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.836+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1780003520 con 0x7f17940b1b10 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3872978927 0 0) 0x7f1780003520 con 0x7f17940b1b10 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 >> v1:172.21.15.18:6790/0 conn(0x7f179406c780 legacy=0x7f17940b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 >> v1:172.21.15.53:6789/0 conn(0x7f1794149f50 legacy=0x7f17940b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 1 monclient: found mon.noname-a 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179406d3e0 con 0x7f17940b1b10 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f178c001a10 con 0x7f17940b1b10 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f179a684640 5 monclient: authenticate success, global_id 4644 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f179a684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f17937fe640 10 monclient: dump: 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:17.856 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:17.857 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.857 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:17.857 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:17.857 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:17.857 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:17.857 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:17.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.855+0000 7f179a684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.862+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f178c001b70 con 0x7f17940b1b10 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.862+0000 7f17937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.862+0000 7f17937fe640 1 -- 172.21.15.18:0/2417700164 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f178c001f80 con 0x7f17940b1b10 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.862+0000 7f17937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.862+0000 7f17937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.862+0000 7f17937fe640 10 monclient: dump: 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.862 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:17.863 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:17.863 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:17.863 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:17.863 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:17.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.862+0000 7f179a684640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:17.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.862+0000 7f179a684640 4 set_mon_vals no callback set 2024-03-25T19:50:17.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.863+0000 7f179a684640 10 monclient: shutdown 2024-03-25T19:50:17.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.863+0000 7f179a684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:17.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.863+0000 7f179a684640 1 -- 172.21.15.18:0/2417700164 >> v1:172.21.15.18:6789/0 conn(0x7f17940b1b10 legacy=0x7f1794152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.863+0000 7f179a684640 1 -- 172.21.15.18:0/2417700164 shutdown_connections 2024-03-25T19:50:17.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.863+0000 7f179a684640 1 -- 172.21.15.18:0/2417700164 >> 172.21.15.18:0/2417700164 conn(0x7f17940b5d10 msgr2=0x7f17940b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:17.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.863+0000 7f179a684640 1 -- 172.21.15.18:0/2417700164 shutdown_connections 2024-03-25T19:50:17.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.863+0000 7f179a684640 1 -- 172.21.15.18:0/2417700164 wait complete. 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 10 monclient: build_initial_monmap 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 10 monclient: monmap: 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 1 librados: starting msgr at 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 1 librados: starting objecter 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 1 Processor -- start 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 1 -- start start 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 1 librados: setting wanted keys 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 1 librados: calling monclient init 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.864+0000 7f179a684640 10 monclient: init 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 10 monclient(hunting): picked mon.noname-c con 0x7f179406c780 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 10 monclient(hunting): picked mon.noname-a con 0x7f17940b1b10 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 10 monclient(hunting): picked mon.noname-b con 0x7f1794149f50 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17941df670 con 0x7f17940b1b10 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17940acfe0 con 0x7f1794149f50 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17941df100 con 0x7f179406c780 2024-03-25T19:50:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:17.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.865+0000 7f179a684640 10 monclient(hunting): authenticate will time out at 5444.992378s 2024-03-25T19:50:17.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.885+0000 7f178bfff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f17940b1b10 0x7f17940aea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41700/0 (socket says 172.21.15.18:41700) 2024-03-25T19:50:17.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.885+0000 7f178bfff640 1 -- 172.21.15.18:0/3890660905 learned_addr learned my addr 172.21.15.18:0/3890660905 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:17.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.887+0000 7f17917fa640 10 client.?.objecter ms_handle_connect 0x7f17940b1b10 2024-03-25T19:50:17.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.887+0000 7f17917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.888+0000 7f17917fa640 10 client.?.objecter ms_handle_connect 0x7f1794149f50 2024-03-25T19:50:17.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.888+0000 7f17917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.891+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795126310 0 0) 0x7f17941df670 con 0x7f17940b1b10 2024-03-25T19:50:17.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.891+0000 7f17917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.891+0000 7f17917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.891+0000 7f17917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.891+0000 7f17917fa640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:50:17.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.891+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1774004eb0 con 0x7f17940b1b10 2024-03-25T19:50:17.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.897+0000 7f17917fa640 10 client.?.objecter ms_handle_connect 0x7f179406c780 2024-03-25T19:50:17.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.897+0000 7f17917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.899+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087501769 0 0) 0x7f17940acfe0 con 0x7f1794149f50 2024-03-25T19:50:17.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.899+0000 7f17917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.899+0000 7f17917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.899+0000 7f17917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.899+0000 7f17917fa640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:50:17.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.899+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17941df670 con 0x7f1794149f50 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.906+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 985414729 0 0) 0x7f1774004eb0 con 0x7f17940b1b10 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.906+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17940acfe0 con 0x7f17940b1b10 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.906+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f177c0019d0 con 0x7f17940b1b10 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.906+0000 7f17917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.906+0000 7f17917fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.906+0000 7f17917fa640 10 monclient(hunting): dump: 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:17.906 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:17.907 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.907 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:17.907 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:17.907 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:17.907 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:17.907 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.906+0000 7f17917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:17.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.914+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1978047474 0 0) 0x7f17941df670 con 0x7f1794149f50 2024-03-25T19:50:17.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.914+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1774004eb0 con 0x7f1794149f50 2024-03-25T19:50:17.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.914+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f178c001ec0 con 0x7f1794149f50 2024-03-25T19:50:17.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.914+0000 7f17917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.914+0000 7f17917fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:17.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.914+0000 7f17917fa640 10 monclient(hunting): dump: 2024-03-25T19:50:17.914 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:17.914 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:17.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2896393509 0 0) 0x7f1774004eb0 con 0x7f1794149f50 2024-03-25T19:50:17.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 >> v1:172.21.15.18:6790/0 conn(0x7f179406c780 legacy=0x7f17941dbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 >> v1:172.21.15.18:6789/0 conn(0x7f17940b1b10 legacy=0x7f17940aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 10 monclient: _finish_hunting 0 2024-03-25T19:50:17.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 1 monclient: found mon.b 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17941df290 con 0x7f1794149f50 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 10 monclient: _finish_auth 0 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 10 monclient: _check_auth_tickets 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f17917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 10 monclient: _renew_subs 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17940af730 con 0x7f1794149f50 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 10 monclient: _renew_subs 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17941484c0 con 0x7f1794149f50 2024-03-25T19:50:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.923+0000 7f179a684640 1 librados: init done 2024-03-25T19:50:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.924+0000 7f179a684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.924+0000 7f179a684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.924+0000 7f179a684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:17.924+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17941df820 con 0x7f1794149f50 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.215+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f178c003c20 con 0x7f1794149f50 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.215+0000 7f17917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.215+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f178c003d90 con 0x7f1794149f50 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.215+0000 7f17917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.215+0000 7f17917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.215+0000 7f17917fa640 10 monclient: dump: 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:18.215 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:18.216 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:18.216 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:18.216 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:18.216 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:18.216 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:18.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.215+0000 7f1792ffd640 4 set_mon_vals no callback set 2024-03-25T19:50:18.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.215+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f178c019d60 con 0x7f1794149f50 2024-03-25T19:50:18.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.216+0000 7f1793fff640 10 monclient: get_auth_request con 0x7f177406de90 auth_method 0 2024-03-25T19:50:18.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.216+0000 7f17917fa640 10 client.4618.objecter ms_handle_connect 0x7f177406de90 2024-03-25T19:50:18.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.223+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f178c053890 con 0x7f1794149f50 2024-03-25T19:50:18.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.223+0000 7f17917fa640 10 client.4618.objecter ms_dispatch 0x7f1794000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:18.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.223+0000 7f17917fa640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:18.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.223+0000 7f17917fa640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:18.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.223+0000 7f17917fa640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T19:50:18.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.223+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f178c04f3a0 con 0x7f1794149f50 2024-03-25T19:50:18.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.223+0000 7f17917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:18.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.223+0000 7f17917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:18.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.483+0000 7f179a684640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.483+0000 7f179a684640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.483+0000 7f179a684640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:18.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.483+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17940627b0 con 0x7f1794149f50 2024-03-25T19:50:18.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.514+0000 7f17917fa640 1 -- 172.21.15.18:0/3890660905 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f178c0512d0 con 0x7f1794149f50 2024-03-25T19:50:18.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.514+0000 7f17917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.514+0000 7f17917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:18.514 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:50:18.515 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:18.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 10 librados: watch_flush enter 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 10 librados: watch_flush exit 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 10 client.4618.objecter successfully canceled tick 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 >> v1:172.21.15.18:6800/1520000573 conn(0x7f177406de90 legacy=0x7f1774070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 10 monclient: shutdown 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 >> v1:172.21.15.53:6789/0 conn(0x7f1794149f50 legacy=0x7f17941e7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 shutdown_connections 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 >> 172.21.15.18:0/3890660905 conn(0x7f17940b5d10 msgr2=0x7f1794143b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 shutdown_connections 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 1 -- 172.21.15.18:0/3890660905 wait complete. 2024-03-25T19:50:18.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:18.517+0000 7f179a684640 1 librados: shutdown 2024-03-25T19:50:18.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:25.335 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.449+0000 7ff3fa2b3640 10 monclient: get_monmap_and_config 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.449+0000 7ff3fa2b3640 10 monclient: build_initial_monmap 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.449+0000 7ff3fa2b3640 10 monclient: monmap: 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:50:25.450441+0000 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:50:25.450441+0000 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:25.449 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:25.450 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:25.450 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:50:25.450 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:50:25.450 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:25.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.450+0000 7ff3fa2b3640 1 Processor -- start 2024-03-25T19:50:25.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.450+0000 7ff3fa2b3640 1 -- start start 2024-03-25T19:50:25.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.450+0000 7ff3fa2b3640 10 monclient: init 2024-03-25T19:50:25.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:25.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:25.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f40a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:25.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f413f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:25.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f41430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:25.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f4142520 con 0x7ff3f41430c0 2024-03-25T19:50:25.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40b28a0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f40b2a30 con 0x7ff3f413f710 2024-03-25T19:50:25.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:25.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.451+0000 7ff3fa2b3640 10 monclient(hunting): authenticate will time out at 5452.578188s 2024-03-25T19:50:25.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.457+0000 7ff3f37fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ff3f413f710 0x7ff3f41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:34868/0 (socket says 172.21.15.18:34868) 2024-03-25T19:50:25.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.457+0000 7ff3f37fe640 1 -- 172.21.15.18:0/820842994 learned_addr learned my addr 172.21.15.18:0/820842994 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:25.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.469+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427572703 0 0) 0x7ff3f40b2a30 con 0x7ff3f413f710 2024-03-25T19:50:25.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.469+0000 7ff3f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.469+0000 7ff3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.469+0000 7ff3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.469+0000 7ff3f2ffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:50:25.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.469+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3dc003520 con 0x7ff3f413f710 2024-03-25T19:50:25.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.477+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813044640 0 0) 0x7ff3f40b28a0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.477+0000 7ff3f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.477+0000 7ff3f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.477+0000 7ff3f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.477+0000 7ff3f2ffd640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:50:25.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.477+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f40b2a30 con 0x7ff3f40a98f0 2024-03-25T19:50:25.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 403318875 0 0) 0x7ff3dc003520 con 0x7ff3f413f710 2024-03-25T19:50:25.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 >> v1:172.21.15.53:6789/0 conn(0x7ff3f40a98f0 legacy=0x7ff3f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 >> v1:172.21.15.18:6789/0 conn(0x7ff3f41430c0 legacy=0x7ff3f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:50:25.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:25.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f40b44c0 con 0x7ff3f413f710 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff3e8001690 con 0x7ff3f413f710 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3f2ffd640 10 monclient: dump: 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.479 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.480 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:25.480 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:25.480 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:25.480 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:25.480 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:25.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3fa2b3640 5 monclient: authenticate success, global_id 4622 2024-03-25T19:50:25.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.478+0000 7ff3fa2b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:25.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3e80017f0 con 0x7ff3f413f710 2024-03-25T19:50:25.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3f2ffd640 1 -- 172.21.15.18:0/820842994 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff3e8001c00 con 0x7ff3f413f710 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3fa2b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3fa2b3640 4 set_mon_vals no callback set 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3f2ffd640 10 monclient: dump: 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:25.488 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:25.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3fa2b3640 10 monclient: shutdown 2024-03-25T19:50:25.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3fa2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:25.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/820842994 >> v1:172.21.15.18:6790/0 conn(0x7ff3f413f710 legacy=0x7ff3f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/820842994 shutdown_connections 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/820842994 >> 172.21.15.18:0/820842994 conn(0x7ff3f40b7ea0 msgr2=0x7ff3f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.488+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/820842994 shutdown_connections 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.489+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/820842994 wait complete. 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.489+0000 7ff3fa2b3640 10 monclient: build_initial_monmap 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.489+0000 7ff3fa2b3640 10 monclient: monmap: 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:50:25.491 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 1 librados: starting msgr at 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 1 librados: starting objecter 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 1 Processor -- start 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 1 -- start start 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 1 librados: setting wanted keys 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 1 librados: calling monclient init 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 10 monclient: init 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.490+0000 7ff3fa2b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.491+0000 7ff3fa2b3640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f40a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.491+0000 7ff3fa2b3640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.491+0000 7ff3fa2b3640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.491+0000 7ff3fa2b3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f4139460 con 0x7ff3f413f710 2024-03-25T19:50:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.491+0000 7ff3fa2b3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f406f1a0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.491+0000 7ff3fa2b3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f41390f0 con 0x7ff3f41430c0 2024-03-25T19:50:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.491+0000 7ff3fa2b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.491+0000 7ff3fa2b3640 10 monclient(hunting): authenticate will time out at 5452.617896s 2024-03-25T19:50:25.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.501+0000 7ff3f3fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff3f40a98f0 0x7ff3f414d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:36080/0 (socket says 172.21.15.18:36080) 2024-03-25T19:50:25.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.501+0000 7ff3f3fff640 1 -- 172.21.15.18:0/2474884339 learned_addr learned my addr 172.21.15.18:0/2474884339 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:25.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.506+0000 7ff3f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff3f40a98f0 2024-03-25T19:50:25.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.506+0000 7ff3f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.506+0000 7ff3f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff3f413f710 2024-03-25T19:50:25.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.506+0000 7ff3f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.510+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732861419 0 0) 0x7ff3f406f1a0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.510+0000 7ff3f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.510+0000 7ff3f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.510+0000 7ff3f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.510+0000 7ff3f0ff9640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:50:25.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.510+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e0004eb0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.511+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357913179 0 0) 0x7ff3f4139460 con 0x7ff3f413f710 2024-03-25T19:50:25.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.511+0000 7ff3f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.511+0000 7ff3f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.511+0000 7ff3f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.511+0000 7ff3f0ff9640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:50:25.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.511+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f406f1a0 con 0x7ff3f413f710 2024-03-25T19:50:25.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.512+0000 7ff3f0ff9640 10 client.?.objecter ms_handle_connect 0x7ff3f41430c0 2024-03-25T19:50:25.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.512+0000 7ff3f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.520+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974477 0 0) 0x7ff3f41390f0 con 0x7ff3f41430c0 2024-03-25T19:50:25.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.520+0000 7ff3f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.520+0000 7ff3f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.520+0000 7ff3f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.520+0000 7ff3f0ff9640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:50:25.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.520+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f4139460 con 0x7ff3f41430c0 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.529+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3773210782 0 0) 0x7ff3e0004eb0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.529+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3f41390f0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.529+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff3e4001a70 con 0x7ff3f40a98f0 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.529+0000 7ff3f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.529+0000 7ff3f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.529+0000 7ff3f0ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.529 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.530 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.530 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:25.530 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:25.530 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:25.530 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:25.530 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.529+0000 7ff3f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.534+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1410965400 0 0) 0x7ff3f4139460 con 0x7ff3f41430c0 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.534+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3e0004eb0 con 0x7ff3f41430c0 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.534+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff3ec0019d0 con 0x7ff3f41430c0 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.534+0000 7ff3f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.534+0000 7ff3f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.534+0000 7ff3f0ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.534 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.535 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:25.535 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:25.535 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:25.535 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:25.535 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:25.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1567567550 0 0) 0x7ff3f41390f0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 >> v1:172.21.15.18:6790/0 conn(0x7ff3f41430c0 legacy=0x7ff3f41e6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 >> v1:172.21.15.18:6789/0 conn(0x7ff3f413f710 legacy=0x7ff3f414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 1 monclient: found mon.b 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f406dea0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 5 monclient: authenticate success, global_id 4624 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 10 monclient: _renew_subs 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f406e150 con 0x7ff3f40a98f0 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 10 monclient: _renew_subs 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f40b7250 con 0x7ff3f40a98f0 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.541+0000 7ff3fa2b3640 1 librados: init done 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.543+0000 7ff3fa2b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.543+0000 7ff3fa2b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.543+0000 7ff3fa2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.543+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f40627b0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.841+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3e4001cd0 con 0x7ff3f40a98f0 2024-03-25T19:50:25.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.841+0000 7ff3f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:25.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.841+0000 7ff3f27fc640 4 set_mon_vals no callback set 2024-03-25T19:50:25.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.841+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff3e4003a40 con 0x7ff3f40a98f0 2024-03-25T19:50:25.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.841+0000 7ff3f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.841+0000 7ff3f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.841+0000 7ff3f0ff9640 10 monclient: dump: 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:25.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.841+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ff3e4003c60 con 0x7ff3f40a98f0 2024-03-25T19:50:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.842+0000 7ff3f37fe640 10 monclient: get_auth_request con 0x7ff3e006df70 auth_method 0 2024-03-25T19:50:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.842+0000 7ff3f0ff9640 10 client.4624.objecter ms_handle_connect 0x7ff3e006df70 2024-03-25T19:50:25.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.847+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7ff3e4054840 con 0x7ff3f40a98f0 2024-03-25T19:50:25.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.847+0000 7ff3f0ff9640 10 client.4624.objecter ms_dispatch 0x7ff3f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:25.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.847+0000 7ff3f0ff9640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:25.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.847+0000 7ff3f0ff9640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:25.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.847+0000 7ff3f0ff9640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T19:50:25.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.847+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff3e4050350 con 0x7ff3f40a98f0 2024-03-25T19:50:25.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.847+0000 7ff3f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:25.847+0000 7ff3f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:26.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.074+0000 7ff3fa2b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.074+0000 7ff3fa2b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.074+0000 7ff3fa2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:50:26.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.074+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3f4069f40 con 0x7ff3f40a98f0 2024-03-25T19:50:26.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.128+0000 7ff3f0ff9640 1 -- 172.21.15.18:0/2474884339 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7ff3e4052280 con 0x7ff3f40a98f0 2024-03-25T19:50:26.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.128+0000 7ff3f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.128+0000 7ff3f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:26.129 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:50:26.129 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:26.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 10 librados: watch_flush enter 2024-03-25T19:50:26.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 10 librados: watch_flush exit 2024-03-25T19:50:26.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T19:50:26.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 10 client.4624.objecter successfully canceled tick 2024-03-25T19:50:26.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 >> v1:172.21.15.18:6800/1520000573 conn(0x7ff3e006df70 legacy=0x7ff3e0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:26.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 10 monclient: shutdown 2024-03-25T19:50:26.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:26.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 >> v1:172.21.15.53:6789/0 conn(0x7ff3f40a98f0 legacy=0x7ff3f414d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:26.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 shutdown_connections 2024-03-25T19:50:26.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.131+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 >> 172.21.15.18:0/2474884339 conn(0x7ff3f40b7ea0 msgr2=0x7ff3f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:26.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.132+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 shutdown_connections 2024-03-25T19:50:26.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.132+0000 7ff3fa2b3640 1 -- 172.21.15.18:0/2474884339 wait complete. 2024-03-25T19:50:26.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:26.132+0000 7ff3fa2b3640 1 librados: shutdown 2024-03-25T19:50:26.143 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:32.950 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.049+0000 7f6277f4d640 10 monclient: get_monmap_and_config 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.049+0000 7f6277f4d640 10 monclient: build_initial_monmap 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.049+0000 7f6277f4d640 10 monclient: monmap: 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:50:33.049945+0000 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:50:33.049945+0000 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.049+0000 7f6277f4d640 1 Processor -- start 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.049+0000 7f6277f4d640 1 -- start start 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.049+0000 7f6277f4d640 10 monclient: init 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 10 monclient(hunting): picked mon.noname-a con 0x7f62700a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 10 monclient(hunting): picked mon.noname-c con 0x7f627013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 10 monclient(hunting): picked mon.noname-b con 0x7f62701430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6270142520 con 0x7f62700a98f0 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62700b28a0 con 0x7f627013f710 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62700b2a30 con 0x7f62701430c0 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.050+0000 7f6277f4d640 10 monclient(hunting): authenticate will time out at 5460.177439s 2024-03-25T19:50:33.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.071+0000 7f62754e8640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f627013f710 0x7f62701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:57772/0 (socket says 172.21.15.18:57772) 2024-03-25T19:50:33.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.071+0000 7f62754e8640 1 -- 172.21.15.18:0/43642358 learned_addr learned my addr 172.21.15.18:0/43642358 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:33.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.086+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896114237 0 0) 0x7f62700b28a0 con 0x7f627013f710 2024-03-25T19:50:33.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.086+0000 7f6274ce7640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.086+0000 7f6274ce7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.086+0000 7f6274ce7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.086+0000 7f6274ce7640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:50:33.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.086+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625c003520 con 0x7f627013f710 2024-03-25T19:50:33.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.094+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691841123 0 0) 0x7f62700b2a30 con 0x7f62701430c0 2024-03-25T19:50:33.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.094+0000 7f6274ce7640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.094+0000 7f6274ce7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.094+0000 7f6274ce7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.094+0000 7f6274ce7640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:50:33.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.094+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62700b28a0 con 0x7f62701430c0 2024-03-25T19:50:33.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347495696 0 0) 0x7f625c003520 con 0x7f627013f710 2024-03-25T19:50:33.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 >> v1:172.21.15.18:6790/0 conn(0x7f62701430c0 legacy=0x7f627014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 >> v1:172.21.15.18:6789/0 conn(0x7f62700a98f0 legacy=0x7f62700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 10 monclient: _finish_hunting 0 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 1 monclient: found mon.noname-c 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62700b44c0 con 0x7f627013f710 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 10 monclient: _finish_auth 0 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 10 monclient: _check_auth_tickets 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6260001690 con 0x7f627013f710 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6274ce7640 10 monclient: dump: 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.096 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.097 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:33.097 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:33.097 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:33.097 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:33.097 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:33.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6277f4d640 5 monclient: authenticate success, global_id 4627 2024-03-25T19:50:33.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.095+0000 7f6277f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.141+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62600017f0 con 0x7f627013f710 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.141+0000 7f6274ce7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.141+0000 7f6274ce7640 1 -- 172.21.15.18:0/43642358 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6260001c00 con 0x7f627013f710 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.141+0000 7f6277f4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.141+0000 7f6277f4d640 4 set_mon_vals no callback set 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.141+0000 7f6274ce7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.141+0000 7f6274ce7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.141+0000 7f6274ce7640 10 monclient: dump: 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.141 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.142 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.142 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:33.142 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:33.142 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:33.142 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:33.142 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.142+0000 7f6277f4d640 10 monclient: shutdown 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.142+0000 7f6277f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.142+0000 7f6277f4d640 1 -- 172.21.15.18:0/43642358 >> v1:172.21.15.53:6789/0 conn(0x7f627013f710 legacy=0x7f62701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.142+0000 7f6277f4d640 1 -- 172.21.15.18:0/43642358 shutdown_connections 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.142+0000 7f6277f4d640 1 -- 172.21.15.18:0/43642358 >> 172.21.15.18:0/43642358 conn(0x7f62700b7ea0 msgr2=0x7f62700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.142+0000 7f6277f4d640 1 -- 172.21.15.18:0/43642358 shutdown_connections 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.142+0000 7f6277f4d640 1 -- 172.21.15.18:0/43642358 wait complete. 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.143+0000 7f6277f4d640 10 monclient: build_initial_monmap 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.143+0000 7f6277f4d640 10 monclient: monmap: 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:50:33.143 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:50:33.144 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:33.144 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:50:33.144 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:33.144 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:50:33.144 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:50:33.144 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:33.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.144+0000 7f6277f4d640 1 librados: starting msgr at 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.144+0000 7f6277f4d640 1 librados: starting objecter 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.144+0000 7f6277f4d640 1 Processor -- start 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.144+0000 7f6277f4d640 1 -- start start 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.144+0000 7f6277f4d640 1 librados: setting wanted keys 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.144+0000 7f6277f4d640 1 librados: calling monclient init 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.144+0000 7f6277f4d640 10 monclient: init 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 10 monclient(hunting): picked mon.noname-c con 0x7f62700a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 10 monclient(hunting): picked mon.noname-b con 0x7f627013f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 10 monclient(hunting): picked mon.noname-a con 0x7f62701430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62701354c0 con 0x7f62701430c0 2024-03-25T19:50:33.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62701319b0 con 0x7f627013f710 2024-03-25T19:50:33.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6270135070 con 0x7f62700a98f0 2024-03-25T19:50:33.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:33.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.145+0000 7f6277f4d640 10 monclient(hunting): authenticate will time out at 5460.272187s 2024-03-25T19:50:33.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.152+0000 7f6275ce9640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f62700a98f0 0x7f62701d7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:44234/0 (socket says 172.21.15.18:44234) 2024-03-25T19:50:33.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.152+0000 7f6275ce9640 1 -- 172.21.15.18:0/2624945851 learned_addr learned my addr 172.21.15.18:0/2624945851 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:33.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.153+0000 7f62667fc640 10 client.?.objecter ms_handle_connect 0x7f62700a98f0 2024-03-25T19:50:33.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.153+0000 7f62667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.159+0000 7f62667fc640 10 client.?.objecter ms_handle_connect 0x7f627013f710 2024-03-25T19:50:33.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.159+0000 7f62667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.161+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464473848 0 0) 0x7f6270135070 con 0x7f62700a98f0 2024-03-25T19:50:33.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.161+0000 7f62667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.161+0000 7f62667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.161+0000 7f62667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.161+0000 7f62667fc640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:50:33.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.162+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6254004eb0 con 0x7f62700a98f0 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 10 client.?.objecter ms_handle_connect 0x7f62701430c0 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 37310032 0 0) 0x7f6254004eb0 con 0x7f62700a98f0 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6270135070 con 0x7f62700a98f0 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f626c0019d0 con 0x7f62700a98f0 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 10 monclient(hunting): dump: 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.176 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:33.176 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:33.176 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:33.176 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:33.176 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:33.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.173+0000 7f62667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:33.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.188+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439883881 0 0) 0x7f62701319b0 con 0x7f627013f710 2024-03-25T19:50:33.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.188+0000 7f62667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.188+0000 7f62667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.188+0000 7f62667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.188+0000 7f62667fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:50:33.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.188+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6254004eb0 con 0x7f627013f710 2024-03-25T19:50:33.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.197+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330906312 0 0) 0x7f62701354c0 con 0x7f62701430c0 2024-03-25T19:50:33.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.197+0000 7f62667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.197+0000 7f62667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.197+0000 7f62667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.197+0000 7f62667fc640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:50:33.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.197+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62701319b0 con 0x7f62701430c0 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3745024458 0 0) 0x7f6254004eb0 con 0x7f627013f710 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62701354c0 con 0x7f627013f710 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6260001b00 con 0x7f627013f710 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 10 monclient(hunting): dump: 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.208 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1798887696 0 0) 0x7f62701319b0 con 0x7f62701430c0 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6254004eb0 con 0x7f62701430c0 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f6268001bd0 con 0x7f62701430c0 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.208+0000 7f62667fc640 10 monclient(hunting): dump: 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:33.209 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.210 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.210 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.210 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:33.210 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:33.210 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:33.210 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:33.210 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1837544332 0 0) 0x7f6270135070 con 0x7f62700a98f0 2024-03-25T19:50:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 >> v1:172.21.15.53:6789/0 conn(0x7f627013f710 legacy=0x7f6270130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 >> v1:172.21.15.18:6789/0 conn(0x7f62701430c0 legacy=0x7f62701312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 1 monclient: found mon.c 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6270135200 con 0x7f62700a98f0 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f62667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 5 monclient: authenticate success, global_id 4631 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 10 monclient: _renew_subs 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62700b7010 con 0x7f62700a98f0 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:33.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 10 monclient: _renew_subs 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f627013b890 con 0x7f62700a98f0 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.210+0000 7f6277f4d640 1 librados: init done 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.211+0000 7f6277f4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.211+0000 7f6277f4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.211+0000 7f6277f4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.211+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6270137160 con 0x7f62700a98f0 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.571+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f626c001c50 con 0x7f62700a98f0 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.571+0000 7f62667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.571+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f626c003a60 con 0x7f62700a98f0 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.571+0000 7f62667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.571+0000 7f62667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.571+0000 7f62667fc640 10 monclient: dump: 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:33.571 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:33.572 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:33.572 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:33.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.571+0000 7f6267fff640 4 set_mon_vals no callback set 2024-03-25T19:50:33.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.571+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f626c003c80 con 0x7f62700a98f0 2024-03-25T19:50:33.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.572+0000 7f62754e8640 10 monclient: get_auth_request con 0x7f625406e1f0 auth_method 0 2024-03-25T19:50:33.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.572+0000 7f62667fc640 10 client.4631.objecter ms_handle_connect 0x7f625406e1f0 2024-03-25T19:50:33.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.580+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f626c054830 con 0x7f62700a98f0 2024-03-25T19:50:33.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.580+0000 7f62667fc640 10 client.4631.objecter ms_dispatch 0x7f6270000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:33.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.580+0000 7f62667fc640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:33.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.580+0000 7f62667fc640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:33.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.580+0000 7f62667fc640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-25T19:50:33.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.580+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f626c050340 con 0x7f62700a98f0 2024-03-25T19:50:33.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.580+0000 7f62667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.580+0000 7f62667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:33.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.825+0000 7f6277f4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.825+0000 7f6277f4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.825+0000 7f6277f4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:33.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.825+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62700627b0 con 0x7f62700a98f0 2024-03-25T19:50:33.842 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:01.372 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.851 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:01.382 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.881 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:01.412 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.900 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:01.397 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.905 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:01.436 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.911 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:01.407 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.934 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:01.465 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.945+0000 7f62667fc640 1 -- 172.21.15.18:0/2624945851 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f626c052270 con 0x7f62700a98f0 2024-03-25T19:50:33.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.945+0000 7f62667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.945+0000 7f62667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:33.945 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:50:33.945 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:33.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 10 librados: watch_flush enter 2024-03-25T19:50:33.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 10 librados: watch_flush exit 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 10 client.4631.objecter successfully canceled tick 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 >> v1:172.21.15.18:6800/1520000573 conn(0x7f625406e1f0 legacy=0x7f62540706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 10 monclient: shutdown 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 >> v1:172.21.15.18:6790/0 conn(0x7f62700a98f0 legacy=0x7f62701d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 shutdown_connections 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 >> 172.21.15.18:0/2624945851 conn(0x7f62700b7ea0 msgr2=0x7f627013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.947+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 shutdown_connections 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.948+0000 7f6277f4d640 1 -- 172.21.15.18:0/2624945851 wait complete. 2024-03-25T19:50:33.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:33.948+0000 7f6277f4d640 1 librados: shutdown 2024-03-25T19:50:33.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:33.959 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:01.454 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.965 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:01.495 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:40.765 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:40.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.873+0000 7fbf0e096640 10 monclient: get_monmap_and_config 2024-03-25T19:50:40.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.873+0000 7fbf0e096640 10 monclient: build_initial_monmap 2024-03-25T19:50:40.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.873+0000 7fbf0e096640 10 monclient: monmap: 2024-03-25T19:50:40.873 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:40.873 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:40.873 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:50:40.874309+0000 2024-03-25T19:50:40.873 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:50:40.874309+0000 2024-03-25T19:50:40.873 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:40.874 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:40.874 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:40.874 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:50:40.874 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:50:40.874 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 1 Processor -- start 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 1 -- start start 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 10 monclient: init 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 10 monclient(hunting): picked mon.noname-c con 0x7fbf080b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 10 monclient(hunting): picked mon.noname-a con 0x7fbf080b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 10 monclient(hunting): picked mon.noname-b con 0x7fbf0814be90 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:40.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf080af940 con 0x7fbf080b1730 2024-03-25T19:50:40.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf080b0a70 con 0x7fbf080b4d60 2024-03-25T19:50:40.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf080b0d50 con 0x7fbf0814be90 2024-03-25T19:50:40.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:40.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.874+0000 7fbf0e096640 10 monclient(hunting): authenticate will time out at 5468.001551s 2024-03-25T19:50:40.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.908+0000 7fbf077fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbf080b4d60 0x7fbf0813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:38482/0 (socket says 172.21.15.18:38482) 2024-03-25T19:50:40.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.908+0000 7fbf077fe640 1 -- 172.21.15.18:0/2734115874 learned_addr learned my addr 172.21.15.18:0/2734115874 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:40.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.935+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193777088 0 0) 0x7fbf080b0d50 con 0x7fbf0814be90 2024-03-25T19:50:40.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.935+0000 7fbf067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.935+0000 7fbf067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.935+0000 7fbf067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.935+0000 7fbf067fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:50:40.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.936+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef0003520 con 0x7fbf0814be90 2024-03-25T19:50:40.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.943+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510427016 0 0) 0x7fbf080b0a70 con 0x7fbf080b4d60 2024-03-25T19:50:40.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.943+0000 7fbf067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.943+0000 7fbf067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.943+0000 7fbf067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.943+0000 7fbf067fc640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:50:40.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.943+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf080b0d50 con 0x7fbf080b4d60 2024-03-25T19:50:40.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3423668226 0 0) 0x7fbef0003520 con 0x7fbf0814be90 2024-03-25T19:50:40.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 >> v1:172.21.15.53:6789/0 conn(0x7fbf080b4d60 legacy=0x7fbf0813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 >> v1:172.21.15.18:6789/0 conn(0x7fbf080b1730 legacy=0x7fbf080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:40.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 1 monclient: found mon.noname-b 2024-03-25T19:50:40.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:40.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:50:40.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf080b1030 con 0x7fbf0814be90 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbf00001a10 con 0x7fbf0814be90 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf067fc640 10 monclient: dump: 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:40.949 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:40.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf0e096640 5 monclient: authenticate success, global_id 4634 2024-03-25T19:50:40.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.948+0000 7fbf0e096640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:40.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.966+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf00001b70 con 0x7fbf0814be90 2024-03-25T19:50:40.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.966+0000 7fbf067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:40.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.966+0000 7fbf067fc640 1 -- 172.21.15.18:0/2734115874 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbf00001f80 con 0x7fbf0814be90 2024-03-25T19:50:40.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.966+0000 7fbf067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.966+0000 7fbf067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.966+0000 7fbf067fc640 10 monclient: dump: 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:40.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.966+0000 7fbf0e096640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:40.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.966+0000 7fbf0e096640 4 set_mon_vals no callback set 2024-03-25T19:50:40.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.967+0000 7fbf0e096640 10 monclient: shutdown 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.967+0000 7fbf0e096640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.967+0000 7fbf0e096640 1 -- 172.21.15.18:0/2734115874 >> v1:172.21.15.18:6790/0 conn(0x7fbf0814be90 legacy=0x7fbf0814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.967+0000 7fbf0e096640 1 -- 172.21.15.18:0/2734115874 shutdown_connections 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.967+0000 7fbf0e096640 1 -- 172.21.15.18:0/2734115874 >> 172.21.15.18:0/2734115874 conn(0x7fbf080617a0 msgr2=0x7fbf08061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.967+0000 7fbf0e096640 1 -- 172.21.15.18:0/2734115874 shutdown_connections 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.967+0000 7fbf0e096640 1 -- 172.21.15.18:0/2734115874 wait complete. 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.968+0000 7fbf0e096640 10 monclient: build_initial_monmap 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.968+0000 7fbf0e096640 10 monclient: monmap: 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:40.970 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.969+0000 7fbf0e096640 1 librados: starting msgr at 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.969+0000 7fbf0e096640 1 librados: starting objecter 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.969+0000 7fbf0e096640 1 Processor -- start 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.969+0000 7fbf0e096640 1 -- start start 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.969+0000 7fbf0e096640 1 librados: setting wanted keys 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.969+0000 7fbf0e096640 1 librados: calling monclient init 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.969+0000 7fbf0e096640 10 monclient: init 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 10 monclient(hunting): picked mon.noname-a con 0x7fbf080b1730 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 10 monclient(hunting): picked mon.noname-c con 0x7fbf080b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:40.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 10 monclient(hunting): picked mon.noname-b con 0x7fbf0814be90 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:40.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf08137b10 con 0x7fbf080b1730 2024-03-25T19:50:40.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf081377e0 con 0x7fbf0814be90 2024-03-25T19:50:40.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf08137fa0 con 0x7fbf080b4d60 2024-03-25T19:50:40.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:40.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.970+0000 7fbf0e096640 10 monclient(hunting): authenticate will time out at 5468.097133s 2024-03-25T19:50:40.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.979+0000 7fbf07fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbf0814be90 0x7fbf081e3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:38510/0 (socket says 172.21.15.18:38510) 2024-03-25T19:50:40.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.979+0000 7fbf07fff640 1 -- 172.21.15.18:0/3871139317 learned_addr learned my addr 172.21.15.18:0/3871139317 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:40.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.982+0000 7fbeeffff640 10 client.?.objecter ms_handle_connect 0x7fbf0814be90 2024-03-25T19:50:40.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.983+0000 7fbeeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.987+0000 7fbeeffff640 10 client.?.objecter ms_handle_connect 0x7fbf080b4d60 2024-03-25T19:50:40.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.987+0000 7fbeeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.992+0000 7fbeeffff640 10 client.?.objecter ms_handle_connect 0x7fbf080b1730 2024-03-25T19:50:40.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.992+0000 7fbeeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.997+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288029351 0 0) 0x7fbf081377e0 con 0x7fbf0814be90 2024-03-25T19:50:40.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.997+0000 7fbeeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.997+0000 7fbeeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.997+0000 7fbeeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.997+0000 7fbeeffff640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:50:40.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:40.997+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef4004eb0 con 0x7fbf0814be90 2024-03-25T19:50:41.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.002+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269973033 0 0) 0x7fbf08137b10 con 0x7fbf080b1730 2024-03-25T19:50:41.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.003+0000 7fbeeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:41.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.003+0000 7fbeeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:41.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.003+0000 7fbeeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:41.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.003+0000 7fbeeffff640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:50:41.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.003+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf081377e0 con 0x7fbf080b1730 2024-03-25T19:50:41.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.013+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834717235 0 0) 0x7fbf08137fa0 con 0x7fbf080b4d60 2024-03-25T19:50:41.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.013+0000 7fbeeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:41.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.013+0000 7fbeeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.013+0000 7fbeeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.013+0000 7fbeeffff640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:50:41.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.013+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf08137b10 con 0x7fbf080b4d60 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.019+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3681032689 0 0) 0x7fbf081377e0 con 0x7fbf080b1730 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.019+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf08137fa0 con 0x7fbf080b1730 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.019+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbef8001930 con 0x7fbf080b1730 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.019+0000 7fbeeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.019+0000 7fbeeffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.019+0000 7fbeeffff640 10 monclient(hunting): dump: 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:41.019 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:41.020 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:41.020 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:41.020 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:41.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.019+0000 7fbeeffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:41.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.029+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2756955652 0 0) 0x7fbef4004eb0 con 0x7fbf0814be90 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.029+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf081377e0 con 0x7fbf0814be90 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.029+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbf00001f40 con 0x7fbf0814be90 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.029+0000 7fbeeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.029+0000 7fbeeffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.029+0000 7fbeeffff640 10 monclient(hunting): dump: 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:41.029 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:41.030 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:41.030 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:41.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3189397035 0 0) 0x7fbf08137fa0 con 0x7fbf080b1730 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 >> v1:172.21.15.18:6790/0 conn(0x7fbf080b4d60 legacy=0x7fbf081daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 >> v1:172.21.15.53:6789/0 conn(0x7fbf0814be90 legacy=0x7fbf081e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 1 monclient: found mon.a 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf08142030 con 0x7fbf080b1730 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 10 monclient: _finish_auth 0 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbeeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 5 monclient: authenticate success, global_id 4662 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 10 monclient: _renew_subs 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:41.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf08140840 con 0x7fbf080b1730 2024-03-25T19:50:41.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 10 client.4662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:41.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 10 monclient: _renew_subs 2024-03-25T19:50:41.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:41.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf081357a0 con 0x7fbf080b1730 2024-03-25T19:50:41.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.038+0000 7fbf0e096640 1 librados: init done 2024-03-25T19:50:41.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.039+0000 7fbf0e096640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:41.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.039+0000 7fbf0e096640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:41.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.039+0000 7fbf0e096640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:41.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.039+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf080627b0 con 0x7fbf080b1730 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.359+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef8001d40 con 0x7fbf080b1730 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.359+0000 7fbeeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.359+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbef8004980 con 0x7fbf080b1730 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.359+0000 7fbeeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.359+0000 7fbeeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.359+0000 7fbeeffff640 10 monclient: dump: 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:41.359 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:41.360 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:41.360 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:41.360 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:41.360 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:41.360 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:41.360 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:41.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.359+0000 7fbf05ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:41.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.359+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fbef8004ba0 con 0x7fbf080b1730 2024-03-25T19:50:41.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.360+0000 7fbf06ffd640 10 monclient: get_auth_request con 0x7fbef406df70 auth_method 0 2024-03-25T19:50:41.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.360+0000 7fbeeffff640 10 client.4662.objecter ms_handle_connect 0x7fbef406df70 2024-03-25T19:50:41.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.380+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7fbef8054720 con 0x7fbf080b1730 2024-03-25T19:50:41.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.380+0000 7fbeeffff640 10 client.4662.objecter ms_dispatch 0x7fbf08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:41.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.380+0000 7fbeeffff640 3 client.4662.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:41.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.380+0000 7fbeeffff640 3 client.4662.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:41.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.380+0000 7fbeeffff640 20 client.4662.objecter dump_active .. 0 homeless 2024-03-25T19:50:41.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.380+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbef8050060 con 0x7fbf080b1730 2024-03-25T19:50:41.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.380+0000 7fbeeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:41.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.380+0000 7fbeeffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:41.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.637+0000 7fbf0e096640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.637+0000 7fbf0e096640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.637+0000 7fbf0e096640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:50:41.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.637+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf080b75b0 con 0x7fbf080b1730 2024-03-25T19:50:41.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.665+0000 7fbeeffff640 1 -- 172.21.15.18:0/3871139317 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7fbef8052160 con 0x7fbf080b1730 2024-03-25T19:50:41.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.665+0000 7fbeeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.665+0000 7fbeeffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:41.665 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:50:41.665 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:41.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 10 librados: watch_flush enter 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 10 librados: watch_flush exit 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 20 client.4662.objecter shutdown clearing up homeless session... 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 10 client.4662.objecter successfully canceled tick 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 >> v1:172.21.15.18:6800/1520000573 conn(0x7fbef406df70 legacy=0x7fbef4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 10 monclient: shutdown 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 >> v1:172.21.15.18:6789/0 conn(0x7fbf080b1730 legacy=0x7fbf08134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 shutdown_connections 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.667+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 >> 172.21.15.18:0/3871139317 conn(0x7fbf080617a0 msgr2=0x7fbf0813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:41.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.668+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 shutdown_connections 2024-03-25T19:50:41.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.668+0000 7fbf0e096640 1 -- 172.21.15.18:0/3871139317 wait complete. 2024-03-25T19:50:41.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:41.668+0000 7fbf0e096640 1 librados: shutdown 2024-03-25T19:50:41.680 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:48.487 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:48.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.602+0000 7f76f077b640 10 monclient: get_monmap_and_config 2024-03-25T19:50:48.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.602+0000 7f76f077b640 10 monclient: build_initial_monmap 2024-03-25T19:50:48.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.602+0000 7f76f077b640 10 monclient: monmap: 2024-03-25T19:50:48.602 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:50:48.603566+0000 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:50:48.603566+0000 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:50:48.603 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:48.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.603+0000 7f76f077b640 1 Processor -- start 2024-03-25T19:50:48.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.603+0000 7f76f077b640 1 -- start start 2024-03-25T19:50:48.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.603+0000 7f76f077b640 10 monclient: init 2024-03-25T19:50:48.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:48.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:48.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 10 monclient(hunting): picked mon.noname-b con 0x7f76e80a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:48.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 10 monclient(hunting): picked mon.noname-a con 0x7f76e813f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:48.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 10 monclient(hunting): picked mon.noname-c con 0x7f76e81430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:48.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e8142520 con 0x7f76e813f710 2024-03-25T19:50:48.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80b28a0 con 0x7f76e81430c0 2024-03-25T19:50:48.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80b2a30 con 0x7f76e80a98f0 2024-03-25T19:50:48.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:48.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.604+0000 7f76f077b640 10 monclient(hunting): authenticate will time out at 5475.731494s 2024-03-25T19:50:48.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.616+0000 7f76edd16640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f76e813f710 0x7f76e81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:49324/0 (socket says 172.21.15.18:49324) 2024-03-25T19:50:48.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.616+0000 7f76edd16640 1 -- 172.21.15.18:0/2536026865 learned_addr learned my addr 172.21.15.18:0/2536026865 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:48.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.628+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504151734 0 0) 0x7f76e8142520 con 0x7f76e813f710 2024-03-25T19:50:48.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.628+0000 7f76ed515640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.628+0000 7f76ed515640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.628+0000 7f76ed515640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.628+0000 7f76ed515640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:50:48.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.628+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d4003520 con 0x7f76e813f710 2024-03-25T19:50:48.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.636+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496727226 0 0) 0x7f76e80b28a0 con 0x7f76e81430c0 2024-03-25T19:50:48.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.636+0000 7f76ed515640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.636+0000 7f76ed515640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.636+0000 7f76ed515640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.636+0000 7f76ed515640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:50:48.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.636+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e8142520 con 0x7f76e81430c0 2024-03-25T19:50:48.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 898603047 0 0) 0x7f76e8142520 con 0x7f76e81430c0 2024-03-25T19:50:48.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 >> v1:172.21.15.18:6790/0 conn(0x7f76e80a98f0 legacy=0x7f76e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 >> v1:172.21.15.18:6789/0 conn(0x7f76e813f710 legacy=0x7f76e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 10 monclient: _finish_hunting 0 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 1 monclient: found mon.noname-c 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e80b44c0 con 0x7f76e81430c0 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 10 monclient: _finish_auth 0 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 10 monclient: _check_auth_tickets 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f76e00019f0 con 0x7f76e81430c0 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76ed515640 10 monclient: dump: 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:48.655 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76f077b640 5 monclient: authenticate success, global_id 4639 2024-03-25T19:50:48.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.654+0000 7f76f077b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.663+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e0001b50 con 0x7f76e81430c0 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.663+0000 7f76ed515640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.663+0000 7f76ed515640 1 -- 172.21.15.18:0/2536026865 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f76e0001fa0 con 0x7f76e81430c0 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.663+0000 7f76ed515640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.663+0000 7f76ed515640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.663+0000 7f76ed515640 10 monclient: dump: 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:48.663 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.664 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:48.664 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:48.664 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:48.664 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:48.664 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:48.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.663+0000 7f76f077b640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:48.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.663+0000 7f76f077b640 4 set_mon_vals no callback set 2024-03-25T19:50:48.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.664+0000 7f76f077b640 10 monclient: shutdown 2024-03-25T19:50:48.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.664+0000 7f76f077b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:48.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.664+0000 7f76f077b640 1 -- 172.21.15.18:0/2536026865 >> v1:172.21.15.53:6789/0 conn(0x7f76e81430c0 legacy=0x7f76e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.664+0000 7f76f077b640 1 -- 172.21.15.18:0/2536026865 shutdown_connections 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.664+0000 7f76f077b640 1 -- 172.21.15.18:0/2536026865 >> 172.21.15.18:0/2536026865 conn(0x7f76e80b7ea0 msgr2=0x7f76e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.664+0000 7f76f077b640 1 -- 172.21.15.18:0/2536026865 shutdown_connections 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.664+0000 7f76f077b640 1 -- 172.21.15.18:0/2536026865 wait complete. 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 10 monclient: build_initial_monmap 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 10 monclient: monmap: 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:50:48.667 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 1 librados: starting msgr at 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 1 librados: starting objecter 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 1 Processor -- start 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 1 -- start start 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 1 librados: setting wanted keys 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 1 librados: calling monclient init 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.665+0000 7f76f077b640 10 monclient: init 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 10 monclient(hunting): picked mon.noname-a con 0x7f76e80a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 10 monclient(hunting): picked mon.noname-c con 0x7f76e813f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 10 monclient(hunting): picked mon.noname-b con 0x7f76e81430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e80abae0 con 0x7f76e80a98f0 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e81d4380 con 0x7f76e81430c0 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e8138f10 con 0x7f76e813f710 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:48.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.666+0000 7f76f077b640 10 monclient(hunting): authenticate will time out at 5475.793428s 2024-03-25T19:50:48.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.671+0000 7f76edd16640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f76e813f710 0x7f76e81d48a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35664/0 (socket says 172.21.15.18:35664) 2024-03-25T19:50:48.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.671+0000 7f76edd16640 1 -- 172.21.15.18:0/2450275560 learned_addr learned my addr 172.21.15.18:0/2450275560 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:48.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.672+0000 7f76deffd640 10 client.?.objecter ms_handle_connect 0x7f76e813f710 2024-03-25T19:50:48.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.672+0000 7f76deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.680+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823258808 0 0) 0x7f76e8138f10 con 0x7f76e813f710 2024-03-25T19:50:48.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.680+0000 7f76deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.680+0000 7f76deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.680+0000 7f76deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.680+0000 7f76deffd640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:50:48.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.680+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc004eb0 con 0x7f76e813f710 2024-03-25T19:50:48.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.687+0000 7f76deffd640 10 client.?.objecter ms_handle_connect 0x7f76e80a98f0 2024-03-25T19:50:48.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.687+0000 7f76deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 998786244 0 0) 0x7f76cc004eb0 con 0x7f76e813f710 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e8138f10 con 0x7f76e813f710 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f76d8001950 con 0x7f76e813f710 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 10 monclient(hunting): dump: 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:48.688 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 10 client.?.objecter ms_handle_connect 0x7f76e81430c0 2024-03-25T19:50:48.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.688+0000 7f76deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.700+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519635480 0 0) 0x7f76e81d4380 con 0x7f76e81430c0 2024-03-25T19:50:48.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.700+0000 7f76deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.700+0000 7f76deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.700+0000 7f76deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.700+0000 7f76deffd640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:50:48.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.700+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc004eb0 con 0x7f76e81430c0 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 320545033 0 0) 0x7f76e8138f10 con 0x7f76e813f710 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 >> v1:172.21.15.53:6789/0 conn(0x7f76e81430c0 legacy=0x7f76e8133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 >> v1:172.21.15.18:6789/0 conn(0x7f76e80a98f0 legacy=0x7f76e8138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 1 monclient: found mon.c 2024-03-25T19:50:48.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e81390a0 con 0x7f76e813f710 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 5 monclient: authenticate success, global_id 4643 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 10 monclient: _renew_subs 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e8135040 con 0x7f76e813f710 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 10 monclient: _renew_subs 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:48.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e81d5190 con 0x7f76e813f710 2024-03-25T19:50:48.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.702+0000 7f76f077b640 1 librados: init done 2024-03-25T19:50:48.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.703+0000 7f76f077b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.703+0000 7f76f077b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.703+0000 7f76f077b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:48.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:48.703+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e80627b0 con 0x7f76e813f710 2024-03-25T19:50:49.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.043+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d80047e0 con 0x7f76e813f710 2024-03-25T19:50:49.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.043+0000 7f76deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:49.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.043+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f76d8004b70 con 0x7f76e813f710 2024-03-25T19:50:49.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.043+0000 7f76deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:49.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.043+0000 7f76deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:49.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.043+0000 7f76deffd640 10 monclient: dump: 2024-03-25T19:50:49.043 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:49.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.043+0000 7f76ecd14640 4 set_mon_vals no callback set 2024-03-25T19:50:49.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.044+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f76d8004d90 con 0x7f76e813f710 2024-03-25T19:50:49.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.045+0000 7f76ee517640 10 monclient: get_auth_request con 0x7f76cc06dd90 auth_method 0 2024-03-25T19:50:49.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.045+0000 7f76deffd640 10 client.4643.objecter ms_handle_connect 0x7f76cc06dd90 2024-03-25T19:50:49.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.051+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3978377181 0 0) 0x7f76d80547f0 con 0x7f76e813f710 2024-03-25T19:50:49.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.051+0000 7f76deffd640 10 client.4643.objecter ms_dispatch 0x7f76e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:49.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.051+0000 7f76deffd640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:49.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.051+0000 7f76deffd640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:49.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.051+0000 7f76deffd640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-25T19:50:49.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.051+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76d8050300 con 0x7f76e813f710 2024-03-25T19:50:49.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.051+0000 7f76deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:49.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.051+0000 7f76deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:49.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.272+0000 7f76f077b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:49.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.272+0000 7f76f077b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:49.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.272+0000 7f76f077b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:49.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.272+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76e81d4510 con 0x7f76e813f710 2024-03-25T19:50:49.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.308+0000 7f76deffd640 1 -- 172.21.15.18:0/2450275560 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544444846) 0x7f76d8052230 con 0x7f76e813f710 2024-03-25T19:50:49.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.308+0000 7f76deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:49.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.308+0000 7f76deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:49.308 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:50:49.308 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:40:55.228039+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:40:55.228039+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 10 librados: watch_flush enter 2024-03-25T19:50:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 10 librados: watch_flush exit 2024-03-25T19:50:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-25T19:50:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 10 client.4643.objecter successfully canceled tick 2024-03-25T19:50:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 >> v1:172.21.15.18:6800/1520000573 conn(0x7f76cc06dd90 legacy=0x7f76cc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 10 monclient: shutdown 2024-03-25T19:50:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 >> v1:172.21.15.18:6790/0 conn(0x7f76e813f710 legacy=0x7f76e81d48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 shutdown_connections 2024-03-25T19:50:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.310+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 >> 172.21.15.18:0/2450275560 conn(0x7f76e80b7ea0 msgr2=0x7f76e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.311+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 shutdown_connections 2024-03-25T19:50:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.311+0000 7f76f077b640 1 -- 172.21.15.18:0/2450275560 wait complete. 2024-03-25T19:50:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:49.311+0000 7f76f077b640 1 librados: shutdown 2024-03-25T19:50:49.321 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:54.060 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:21.590 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:54.074 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:21.605 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:54.148 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:21.678 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:54.192 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:21.688 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:54.196 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:21.727 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:54.203 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:21.734 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:54.205 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:21.702 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:54.232 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:21.728 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:54.244 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:21.775 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:56.127 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:56.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.245+0000 7faa19c7b640 10 monclient: get_monmap_and_config 2024-03-25T19:50:56.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.245+0000 7faa19c7b640 10 monclient: build_initial_monmap 2024-03-25T19:50:56.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.245+0000 7faa19c7b640 10 monclient: monmap: 2024-03-25T19:50:56.245 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:56.245 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:56.245 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:50:56.246563+0000 2024-03-25T19:50:56.245 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:50:56.246563+0000 2024-03-25T19:50:56.246 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:56.246 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:56.246 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:56.246 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:50:56.246 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:50:56.246 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:56.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.246+0000 7faa19c7b640 1 Processor -- start 2024-03-25T19:50:56.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.246+0000 7faa19c7b640 1 -- start start 2024-03-25T19:50:56.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.246+0000 7faa19c7b640 10 monclient: init 2024-03-25T19:50:56.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:56.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:56.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 10 monclient(hunting): picked mon.noname-a con 0x7faa140b10d0 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:56.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 10 monclient(hunting): picked mon.noname-c con 0x7faa140b4360 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:56.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 10 monclient(hunting): picked mon.noname-b con 0x7faa140a9470 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:56.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa140a9ab0 con 0x7faa140b10d0 2024-03-25T19:50:56.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa140b2400 con 0x7faa140b4360 2024-03-25T19:50:56.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa140aa2a0 con 0x7faa140a9470 2024-03-25T19:50:56.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:56.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.247+0000 7faa19c7b640 10 monclient(hunting): authenticate will time out at 5483.374494s 2024-03-25T19:50:56.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.258+0000 7faa12ffd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faa140b4360 0x7faa140b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:44582/0 (socket says 172.21.15.18:44582) 2024-03-25T19:50:56.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.258+0000 7faa12ffd640 1 -- 172.21.15.18:0/2573216725 learned_addr learned my addr 172.21.15.18:0/2573216725 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:56.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.279+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443972297 0 0) 0x7faa140aa2a0 con 0x7faa140a9470 2024-03-25T19:50:56.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.279+0000 7faa127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.279+0000 7faa127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.279+0000 7faa127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.279+0000 7faa127fc640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:50:56.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.279+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc003520 con 0x7faa140a9470 2024-03-25T19:50:56.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.284+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024511689 0 0) 0x7faa140b2400 con 0x7faa140b4360 2024-03-25T19:50:56.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.284+0000 7faa127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.284+0000 7faa127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.284+0000 7faa127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.284+0000 7faa127fc640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:50:56.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.284+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa140aa2a0 con 0x7faa140b4360 2024-03-25T19:50:56.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.287+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929916410 0 0) 0x7faa140a9ab0 con 0x7faa140b10d0 2024-03-25T19:50:56.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.287+0000 7faa127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.287+0000 7faa127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.287+0000 7faa127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.287+0000 7faa127fc640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:50:56.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.287+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa140b2400 con 0x7faa140b10d0 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3543373076 0 0) 0x7fa9fc003520 con 0x7faa140a9470 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 >> v1:172.21.15.53:6789/0 conn(0x7faa140b4360 legacy=0x7faa140b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 >> v1:172.21.15.18:6789/0 conn(0x7faa140b10d0 legacy=0x7faa140b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 1 monclient: found mon.noname-b 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa140b14b0 con 0x7faa140a9470 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:56.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faa0c001a10 con 0x7faa140a9470 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa19c7b640 5 monclient: authenticate success, global_id 4646 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa19c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa127fc640 10 monclient: dump: 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:56.293 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:56.294 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:56.294 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:56.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.292+0000 7faa19c7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.303+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa0c001b70 con 0x7faa140a9470 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.303+0000 7faa127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.303+0000 7faa127fc640 1 -- 172.21.15.18:0/2573216725 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faa0c001f80 con 0x7faa140a9470 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.303+0000 7faa127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.303+0000 7faa127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.303+0000 7faa127fc640 10 monclient: dump: 2024-03-25T19:50:56.303 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.303+0000 7faa19c7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:56.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.303+0000 7faa19c7b640 4 set_mon_vals no callback set 2024-03-25T19:50:56.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.304+0000 7faa19c7b640 10 monclient: shutdown 2024-03-25T19:50:56.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.304+0000 7faa19c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:56.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.304+0000 7faa19c7b640 1 -- 172.21.15.18:0/2573216725 >> v1:172.21.15.18:6790/0 conn(0x7faa140a9470 legacy=0x7faa1414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.304+0000 7faa19c7b640 1 -- 172.21.15.18:0/2573216725 shutdown_connections 2024-03-25T19:50:56.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.304+0000 7faa19c7b640 1 -- 172.21.15.18:0/2573216725 >> 172.21.15.18:0/2573216725 conn(0x7faa140617a0 msgr2=0x7faa14061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:56.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.304+0000 7faa19c7b640 1 -- 172.21.15.18:0/2573216725 shutdown_connections 2024-03-25T19:50:56.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.304+0000 7faa19c7b640 1 -- 172.21.15.18:0/2573216725 wait complete. 2024-03-25T19:50:56.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.305+0000 7faa19c7b640 10 monclient: build_initial_monmap 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.305+0000 7faa19c7b640 10 monclient: monmap: 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.305+0000 7faa19c7b640 1 librados: starting msgr at 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.305+0000 7faa19c7b640 1 librados: starting objecter 2024-03-25T19:50:56.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.305+0000 7faa19c7b640 1 Processor -- start 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 1 -- start start 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 1 librados: setting wanted keys 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 1 librados: calling monclient init 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 10 monclient: init 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 10 monclient(hunting): picked mon.noname-c con 0x7faa140a9470 addr v1:172.21.15.18:6790/0 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 10 monclient(hunting): picked mon.noname-b con 0x7faa140b10d0 addr v1:172.21.15.53:6789/0 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 10 monclient(hunting): picked mon.noname-a con 0x7faa140b4360 addr v1:172.21.15.18:6789/0 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa140adb40 con 0x7faa140b4360 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa140ade20 con 0x7faa140b10d0 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa140ae100 con 0x7faa140a9470 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:56.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.306+0000 7faa19c7b640 10 monclient(hunting): authenticate will time out at 5483.433520s 2024-03-25T19:50:56.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.312+0000 7faa137fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7faa140a9470 0x7faa1413e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:43196/0 (socket says 172.21.15.18:43196) 2024-03-25T19:50:56.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.312+0000 7faa137fe640 1 -- 172.21.15.18:0/557518255 learned_addr learned my addr 172.21.15.18:0/557518255 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:50:56.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.320+0000 7fa9fbfff640 10 client.?.objecter ms_handle_connect 0x7faa140a9470 2024-03-25T19:50:56.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.320+0000 7fa9fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.320+0000 7fa9fbfff640 10 client.?.objecter ms_handle_connect 0x7faa140b4360 2024-03-25T19:50:56.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.320+0000 7fa9fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.328+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692149446 0 0) 0x7faa140ae100 con 0x7faa140a9470 2024-03-25T19:50:56.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.328+0000 7fa9fbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.328+0000 7fa9fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.328+0000 7fa9fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.328+0000 7fa9fbfff640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:50:56.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.328+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa00004eb0 con 0x7faa140a9470 2024-03-25T19:50:56.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.333+0000 7fa9fbfff640 10 client.?.objecter ms_handle_connect 0x7faa140b10d0 2024-03-25T19:50:56.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.333+0000 7fa9fbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.345+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146264365 0 0) 0x7faa140adb40 con 0x7faa140b4360 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.345+0000 7fa9fbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.345+0000 7fa9fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.345+0000 7fa9fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.345+0000 7fa9fbfff640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:50:56.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.345+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa140ae100 con 0x7faa140b4360 2024-03-25T19:50:56.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026436549 0 0) 0x7faa140ade20 con 0x7faa140b10d0 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa140adb40 con 0x7faa140b10d0 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2698799043 0 0) 0x7faa00004eb0 con 0x7faa140a9470 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa140ade20 con 0x7faa140a9470 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faa04001a10 con 0x7faa140a9470 2024-03-25T19:50:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 10 monclient(hunting): dump: 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:56.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.347+0000 7fa9fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:56.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2314181782 0 0) 0x7faa140ade20 con 0x7faa140a9470 2024-03-25T19:50:56.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 >> v1:172.21.15.53:6789/0 conn(0x7faa140b10d0 legacy=0x7faa140ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 >> v1:172.21.15.18:6789/0 conn(0x7faa140b4360 legacy=0x7faa141e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 1 monclient: found mon.c 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa140acd20 con 0x7faa140a9470 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 10 monclient: _finish_auth 0 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7fa9fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 10 monclient: _renew_subs 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa140abaa0 con 0x7faa140a9470 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:56.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 10 monclient: _renew_subs 2024-03-25T19:50:56.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:56.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa140ac900 con 0x7faa140a9470 2024-03-25T19:50:56.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.358+0000 7faa19c7b640 1 librados: init done 2024-03-25T19:50:56.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.359+0000 7faa19c7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.359+0000 7faa19c7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.359+0000 7faa19c7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:56.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.359+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa140abda0 con 0x7faa140a9470 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.664+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa04001c70 con 0x7faa140a9470 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.664+0000 7fa9fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.664+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faa040038c0 con 0x7faa140a9470 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.664+0000 7fa9fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.664+0000 7fa9fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.664+0000 7fa9fbfff640 10 monclient: dump: 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:50:56.664 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:56.665 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:50:56.665 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.665 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:50:56.665 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:50:56.665 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:50:56.665 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:50:56.665 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:50:56.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.664+0000 7faa11ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:56.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.665+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7faa04003ae0 con 0x7faa140a9470 2024-03-25T19:50:56.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.665+0000 7faa12ffd640 10 monclient: get_auth_request con 0x7faa0006ddf0 auth_method 0 2024-03-25T19:50:56.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.666+0000 7fa9fbfff640 10 client.4649.objecter ms_handle_connect 0x7faa0006ddf0 2024-03-25T19:50:56.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.671+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7faa04004460 con 0x7faa140a9470 2024-03-25T19:50:56.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.671+0000 7fa9fbfff640 10 client.4649.objecter ms_dispatch 0x7faa14000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:50:56.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.671+0000 7fa9fbfff640 3 client.4649.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:50:56.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.671+0000 7fa9fbfff640 3 client.4649.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:50:56.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.671+0000 7fa9fbfff640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T19:50:56.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.671+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa040501a0 con 0x7faa140a9470 2024-03-25T19:50:56.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.671+0000 7fa9fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.671+0000 7fa9fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:56.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.920+0000 7faa19c7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.920+0000 7faa19c7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.920+0000 7faa19c7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:50:56.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:56.921+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa140aceb0 con 0x7faa140a9470 2024-03-25T19:50:57.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.026+0000 7fa9fbfff640 1 -- 172.21.15.18:0/557518255 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7faa040520d0 con 0x7faa140a9470 2024-03-25T19:50:57.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.026+0000 7fa9fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:57.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.026+0000 7fa9fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:57.027 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:50:57.027 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:57.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 10 librados: watch_flush enter 2024-03-25T19:50:57.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 10 librados: watch_flush exit 2024-03-25T19:50:57.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T19:50:57.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 10 client.4649.objecter successfully canceled tick 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 >> v1:172.21.15.18:6800/1520000573 conn(0x7faa0006ddf0 legacy=0x7faa000702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 10 monclient: shutdown 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 >> v1:172.21.15.18:6790/0 conn(0x7faa140a9470 legacy=0x7faa1413e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 shutdown_connections 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 >> 172.21.15.18:0/557518255 conn(0x7faa140617a0 msgr2=0x7faa14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 shutdown_connections 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 1 -- 172.21.15.18:0/557518255 wait complete. 2024-03-25T19:50:57.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:50:57.029+0000 7faa19c7b640 1 librados: shutdown 2024-03-25T19:50:57.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:03.847 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:03.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.945+0000 7f9d72d32640 10 monclient: get_monmap_and_config 2024-03-25T19:51:03.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.945+0000 7f9d72d32640 10 monclient: build_initial_monmap 2024-03-25T19:51:03.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.945+0000 7f9d72d32640 10 monclient: monmap: 2024-03-25T19:51:03.945 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:51:03.946665+0000 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:51:03.946665+0000 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:51:03.946 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:03.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.946+0000 7f9d72d32640 1 Processor -- start 2024-03-25T19:51:03.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.946+0000 7f9d72d32640 1 -- start start 2024-03-25T19:51:03.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.946+0000 7f9d72d32640 10 monclient: init 2024-03-25T19:51:03.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:03.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:03.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 10 monclient(hunting): picked mon.noname-a con 0x7f9d6c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:03.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 10 monclient(hunting): picked mon.noname-c con 0x7f9d6c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:03.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:03.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c142520 con 0x7f9d6c0a98f0 2024-03-25T19:51:03.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c0b28a0 con 0x7f9d6c13f710 2024-03-25T19:51:03.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c0b2a30 con 0x7f9d6c1430c0 2024-03-25T19:51:03.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:03.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.947+0000 7f9d72d32640 10 monclient(hunting): authenticate will time out at 5491.074611s 2024-03-25T19:51:03.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.954+0000 7f9d70ace640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f9d6c0a98f0 0x7f9d6c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41884/0 (socket says 172.21.15.18:41884) 2024-03-25T19:51:03.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.954+0000 7f9d70ace640 1 -- 172.21.15.18:0/3259774113 learned_addr learned my addr 172.21.15.18:0/3259774113 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:03.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.962+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988270414 0 0) 0x7f9d6c142520 con 0x7f9d6c0a98f0 2024-03-25T19:51:03.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.962+0000 7f9d6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.962+0000 7f9d6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.962+0000 7f9d6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.962+0000 7f9d6b7fe640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:51:03.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.962+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d5c003520 con 0x7f9d6c0a98f0 2024-03-25T19:51:03.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1282997438 0 0) 0x7f9d5c003520 con 0x7f9d6c0a98f0 2024-03-25T19:51:03.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 >> v1:172.21.15.18:6790/0 conn(0x7f9d6c1430c0 legacy=0x7f9d6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:51:03.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 >> v1:172.21.15.53:6789/0 conn(0x7f9d6c13f710 legacy=0x7f9d6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:03.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c0b44c0 con 0x7f9d6c0a98f0 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9d60001690 con 0x7f9d6c0a98f0 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d6b7fe640 10 monclient: dump: 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:03.979 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.980 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:03.980 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:03.980 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:03.980 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:03.980 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:03.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d72d32640 5 monclient: authenticate success, global_id 4677 2024-03-25T19:51:03.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.978+0000 7f9d72d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:03.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.997+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d60001860 con 0x7f9d6c0a98f0 2024-03-25T19:51:03.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.997+0000 7f9d6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:03.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.997+0000 7f9d6b7fe640 1 -- 172.21.15.18:0/3259774113 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9d60001cb0 con 0x7f9d6c0a98f0 2024-03-25T19:51:03.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.997+0000 7f9d6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.997+0000 7f9d6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:03.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.997+0000 7f9d6b7fe640 10 monclient: dump: 2024-03-25T19:51:03.997 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:03.997 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.997+0000 7f9d72d32640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:03.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.997+0000 7f9d72d32640 4 set_mon_vals no callback set 2024-03-25T19:51:04.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.998+0000 7f9d72d32640 10 monclient: shutdown 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.998+0000 7f9d72d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.998+0000 7f9d72d32640 1 -- 172.21.15.18:0/3259774113 >> v1:172.21.15.18:6789/0 conn(0x7f9d6c0a98f0 legacy=0x7f9d6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.998+0000 7f9d72d32640 1 -- 172.21.15.18:0/3259774113 shutdown_connections 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.998+0000 7f9d72d32640 1 -- 172.21.15.18:0/3259774113 >> 172.21.15.18:0/3259774113 conn(0x7f9d6c0b7ea0 msgr2=0x7f9d6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.998+0000 7f9d72d32640 1 -- 172.21.15.18:0/3259774113 shutdown_connections 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.998+0000 7f9d72d32640 1 -- 172.21.15.18:0/3259774113 wait complete. 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 10 monclient: build_initial_monmap 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 10 monclient: monmap: 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:04.001 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 1 librados: starting msgr at 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 1 librados: starting objecter 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 1 Processor -- start 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 1 -- start start 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 1 librados: setting wanted keys 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 1 librados: calling monclient init 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:03.999+0000 7f9d72d32640 10 monclient: init 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.000+0000 7f9d72d32640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.000+0000 7f9d72d32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.000+0000 7f9d72d32640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.000+0000 7f9d72d32640 10 monclient(hunting): picked mon.noname-a con 0x7f9d6c13f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.000+0000 7f9d72d32640 10 monclient(hunting): picked mon.noname-c con 0x7f9d6c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.000+0000 7f9d72d32640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c0ab8b0 con 0x7f9d6c13f710 2024-03-25T19:51:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.000+0000 7f9d72d32640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c138740 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.001+0000 7f9d72d32640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c134690 con 0x7f9d6c1430c0 2024-03-25T19:51:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.001+0000 7f9d72d32640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.001+0000 7f9d72d32640 10 monclient(hunting): authenticate will time out at 5491.127836s 2024-03-25T19:51:04.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.009+0000 7f9d70ace640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9d6c0a98f0 0x7f9d6c1377d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:49394/0 (socket says 172.21.15.18:49394) 2024-03-25T19:51:04.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.009+0000 7f9d70ace640 1 -- 172.21.15.18:0/1389757780 learned_addr learned my addr 172.21.15.18:0/1389757780 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:04.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.011+0000 7f9d697fa640 10 client.?.objecter ms_handle_connect 0x7f9d6c0a98f0 2024-03-25T19:51:04.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.011+0000 7f9d697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:04.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.014+0000 7f9d697fa640 10 client.?.objecter ms_handle_connect 0x7f9d6c13f710 2024-03-25T19:51:04.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.014+0000 7f9d697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:04.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.019+0000 7f9d697fa640 10 client.?.objecter ms_handle_connect 0x7f9d6c1430c0 2024-03-25T19:51:04.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.019+0000 7f9d697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.022+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272017755 0 0) 0x7f9d6c138740 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.022+0000 7f9d697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.022+0000 7f9d697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.022+0000 7f9d697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.022+0000 7f9d697fa640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:51:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.022+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d50004eb0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.025+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831114098 0 0) 0x7f9d6c0ab8b0 con 0x7f9d6c13f710 2024-03-25T19:51:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.025+0000 7f9d697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.025+0000 7f9d697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.025+0000 7f9d697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.025+0000 7f9d697fa640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:51:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.025+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d6c138740 con 0x7f9d6c13f710 2024-03-25T19:51:04.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.033+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 986743653 0 0) 0x7f9d50004eb0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.033+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d6c0ab8b0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.033+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9d60001b70 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.033+0000 7f9d697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:04.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.033+0000 7f9d697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:04.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.033+0000 7f9d697fa640 10 monclient(hunting): dump: 2024-03-25T19:51:04.033 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:04.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.033+0000 7f9d697fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:04.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.034+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284565183 0 0) 0x7f9d6c134690 con 0x7f9d6c1430c0 2024-03-25T19:51:04.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.034+0000 7f9d697fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:04.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.034+0000 7f9d697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:04.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.034+0000 7f9d697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:04.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.034+0000 7f9d697fa640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:51:04.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.034+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d50004eb0 con 0x7f9d6c1430c0 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.038+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1331331176 0 0) 0x7f9d6c138740 con 0x7f9d6c13f710 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.038+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d6c134690 con 0x7f9d6c13f710 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.038+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9d580018b0 con 0x7f9d6c13f710 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.038+0000 7f9d697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.038+0000 7f9d697fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.038+0000 7f9d697fa640 10 monclient(hunting): dump: 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:04.038 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:04.039 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:04.039 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:04.039 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:04.039 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:04.039 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:04.039 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:04.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1093068595 0 0) 0x7f9d6c0ab8b0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 >> v1:172.21.15.18:6790/0 conn(0x7f9d6c1430c0 legacy=0x7f9d6c1da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 >> v1:172.21.15.18:6789/0 conn(0x7f9d6c13f710 legacy=0x7f9d6c133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 1 monclient: found mon.b 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c134820 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 5 monclient: authenticate success, global_id 4654 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 10 monclient: _renew_subs 2024-03-25T19:51:04.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6c137fe0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 10 monclient: _renew_subs 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c1384c0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.043+0000 7f9d72d32640 1 librados: init done 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.045+0000 7f9d72d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.045+0000 7f9d72d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:04.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.045+0000 7f9d72d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:04.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.045+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c0b3ef0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.341+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d60004bf0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.341+0000 7f9d697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:04.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.341+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f9d60004f00 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.341+0000 7f9d697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:04.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.341+0000 7f9d697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:04.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.341+0000 7f9d697fa640 10 monclient: dump: 2024-03-25T19:51:04.341 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:04.341 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:04.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.341+0000 7f9d6affd640 4 set_mon_vals no callback set 2024-03-25T19:51:04.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.342+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f9d6001aed0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.342+0000 7f9d6bfff640 10 monclient: get_auth_request con 0x7f9d5006df60 auth_method 0 2024-03-25T19:51:04.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.343+0000 7f9d697fa640 10 client.4654.objecter ms_handle_connect 0x7f9d5006df60 2024-03-25T19:51:04.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.349+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7f9d60054960 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.349+0000 7f9d697fa640 10 client.4654.objecter ms_dispatch 0x7f9d6c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:04.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.349+0000 7f9d697fa640 3 client.4654.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:04.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.349+0000 7f9d697fa640 3 client.4654.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:04.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.349+0000 7f9d697fa640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T19:51:04.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.349+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d60050470 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.349+0000 7f9d697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:04.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.349+0000 7f9d697fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:04.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.601+0000 7f9d72d32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.601+0000 7f9d72d32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.601+0000 7f9d72d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:04.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.601+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d6c1388d0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.630+0000 7f9d697fa640 1 -- 172.21.15.18:0/1389757780 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7f9d600523a0 con 0x7f9d6c0a98f0 2024-03-25T19:51:04.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.630+0000 7f9d697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.630+0000 7f9d697fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:04.630 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:51:04.630 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:04.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.632+0000 7f9d72d32640 10 librados: watch_flush enter 2024-03-25T19:51:04.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 10 librados: watch_flush exit 2024-03-25T19:51:04.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T19:51:04.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 10 client.4654.objecter successfully canceled tick 2024-03-25T19:51:04.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 >> v1:172.21.15.18:6800/1520000573 conn(0x7f9d5006df60 legacy=0x7f9d50070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 10 monclient: shutdown 2024-03-25T19:51:04.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:04.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 >> v1:172.21.15.53:6789/0 conn(0x7f9d6c0a98f0 legacy=0x7f9d6c1377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 shutdown_connections 2024-03-25T19:51:04.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 >> 172.21.15.18:0/1389757780 conn(0x7f9d6c0b7ea0 msgr2=0x7f9d6c13d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:04.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 shutdown_connections 2024-03-25T19:51:04.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 1 -- 172.21.15.18:0/1389757780 wait complete. 2024-03-25T19:51:04.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:04.633+0000 7f9d72d32640 1 librados: shutdown 2024-03-25T19:51:04.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:11.452 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.568+0000 7faabed82640 10 monclient: get_monmap_and_config 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.568+0000 7faabed82640 10 monclient: build_initial_monmap 2024-03-25T19:51:11.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.568+0000 7faabed82640 10 monclient: monmap: 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:51:11.569789+0000 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:51:11.569789+0000 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:51:11.569 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.569+0000 7faabed82640 1 Processor -- start 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.570+0000 7faabed82640 1 -- start start 2024-03-25T19:51:11.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.570+0000 7faabed82640 10 monclient: init 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.570+0000 7faabed82640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.570+0000 7faabed82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.570+0000 7faabed82640 10 monclient(hunting): picked mon.noname-a con 0x7faab80a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.571+0000 7faabed82640 10 monclient(hunting): picked mon.noname-c con 0x7faab813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.571+0000 7faabed82640 10 monclient(hunting): picked mon.noname-b con 0x7faab81430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.571+0000 7faabed82640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab8142520 con 0x7faab80a98f0 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.571+0000 7faabed82640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab80b28a0 con 0x7faab813f710 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.571+0000 7faabed82640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab80b2a30 con 0x7faab81430c0 2024-03-25T19:51:11.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.571+0000 7faabed82640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:11.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.571+0000 7faabed82640 10 monclient(hunting): authenticate will time out at 5498.697892s 2024-03-25T19:51:11.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.574+0000 7faab7fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faab813f710 0x7faab81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:42664/0 (socket says 172.21.15.18:42664) 2024-03-25T19:51:11.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.574+0000 7faab7fff640 1 -- 172.21.15.18:0/159517377 learned_addr learned my addr 172.21.15.18:0/159517377 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:11.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.581+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749417569 0 0) 0x7faab80b28a0 con 0x7faab813f710 2024-03-25T19:51:11.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.581+0000 7faab77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.581+0000 7faab77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.581+0000 7faab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.581+0000 7faab77fe640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:51:11.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.581+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa8003520 con 0x7faab813f710 2024-03-25T19:51:11.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.588+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998713391 0 0) 0x7faab80b2a30 con 0x7faab81430c0 2024-03-25T19:51:11.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.588+0000 7faab77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.588+0000 7faab77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.588+0000 7faab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.588+0000 7faab77fe640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:51:11.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.588+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab80b28a0 con 0x7faab81430c0 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2303960502 0 0) 0x7faaa8003520 con 0x7faab813f710 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 >> v1:172.21.15.18:6790/0 conn(0x7faab81430c0 legacy=0x7faab814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 >> v1:172.21.15.18:6789/0 conn(0x7faab80a98f0 legacy=0x7faab80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 1 monclient: found mon.noname-c 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab80b44c0 con 0x7faab813f710 2024-03-25T19:51:11.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faaa4001690 con 0x7faab813f710 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faab77fe640 10 monclient: dump: 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:11.593 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faabed82640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:51:11.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.592+0000 7faabed82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.604+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaa40017f0 con 0x7faab813f710 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.604+0000 7faab77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.604+0000 7faab77fe640 1 -- 172.21.15.18:0/159517377 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faaa4001c00 con 0x7faab813f710 2024-03-25T19:51:11.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.604+0000 7faab77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.604+0000 7faab77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.604+0000 7faab77fe640 10 monclient: dump: 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.604+0000 7faabed82640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:11.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.604+0000 7faabed82640 4 set_mon_vals no callback set 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 10 monclient: shutdown 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 1 -- 172.21.15.18:0/159517377 >> v1:172.21.15.53:6789/0 conn(0x7faab813f710 legacy=0x7faab81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 1 -- 172.21.15.18:0/159517377 shutdown_connections 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 1 -- 172.21.15.18:0/159517377 >> 172.21.15.18:0/159517377 conn(0x7faab80b7ea0 msgr2=0x7faab80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 1 -- 172.21.15.18:0/159517377 shutdown_connections 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 1 -- 172.21.15.18:0/159517377 wait complete. 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 10 monclient: build_initial_monmap 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.605+0000 7faabed82640 10 monclient: monmap: 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:11.606 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 1 librados: starting msgr at 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 1 librados: starting objecter 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 1 Processor -- start 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 1 -- start start 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 1 librados: setting wanted keys 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 1 librados: calling monclient init 2024-03-25T19:51:11.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 10 monclient: init 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 10 monclient(hunting): picked mon.noname-a con 0x7faab80a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 10 monclient(hunting): picked mon.noname-c con 0x7faab813f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 10 monclient(hunting): picked mon.noname-b con 0x7faab81430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.606+0000 7faabed82640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab8133a40 con 0x7faab80a98f0 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.607+0000 7faabed82640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab8138a70 con 0x7faab81430c0 2024-03-25T19:51:11.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.607+0000 7faabed82640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab81379f0 con 0x7faab813f710 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.607+0000 7faabed82640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:11.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.607+0000 7faabed82640 10 monclient(hunting): authenticate will time out at 5498.733772s 2024-03-25T19:51:11.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.618+0000 7faab7fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7faab813f710 0x7faab8132c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41968/0 (socket says 172.21.15.18:41968) 2024-03-25T19:51:11.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.618+0000 7faab7fff640 1 -- 172.21.15.18:0/2572666147 learned_addr learned my addr 172.21.15.18:0/2572666147 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:11.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.619+0000 7faab57fa640 10 client.?.objecter ms_handle_connect 0x7faab813f710 2024-03-25T19:51:11.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.619+0000 7faab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.621+0000 7faab57fa640 10 client.?.objecter ms_handle_connect 0x7faab80a98f0 2024-03-25T19:51:11.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.621+0000 7faab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.626+0000 7faab57fa640 10 client.?.objecter ms_handle_connect 0x7faab81430c0 2024-03-25T19:51:11.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.626+0000 7faab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.629+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164192005 0 0) 0x7faab8133a40 con 0x7faab80a98f0 2024-03-25T19:51:11.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.629+0000 7faab57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.629+0000 7faab57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.629+0000 7faab57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.629+0000 7faab57fa640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:51:11.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.629+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa9c004eb0 con 0x7faab80a98f0 2024-03-25T19:51:11.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.640+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016331200 0 0) 0x7faab8138a70 con 0x7faab81430c0 2024-03-25T19:51:11.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.640+0000 7faab57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.640+0000 7faab57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.640+0000 7faab57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.640+0000 7faab57fa640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:51:11.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.640+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab8133a40 con 0x7faab81430c0 2024-03-25T19:51:11.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.642+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2110162643 0 0) 0x7faa9c004eb0 con 0x7faab80a98f0 2024-03-25T19:51:11.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.642+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faab8138a70 con 0x7faab80a98f0 2024-03-25T19:51:11.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.642+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faaac001870 con 0x7faab80a98f0 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.642+0000 7faab57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.642+0000 7faab57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.642+0000 7faab57fa640 10 monclient(hunting): dump: 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:11.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.642+0000 7faab57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:11.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.645+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907986695 0 0) 0x7faab81379f0 con 0x7faab813f710 2024-03-25T19:51:11.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.645+0000 7faab57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.645+0000 7faab57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.645+0000 7faab57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.645+0000 7faab57fa640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:51:11.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.645+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa9c004eb0 con 0x7faab813f710 2024-03-25T19:51:11.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.658+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3674907681 0 0) 0x7faab8133a40 con 0x7faab81430c0 2024-03-25T19:51:11.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.658+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faab81379f0 con 0x7faab81430c0 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.658+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faab0001bd0 con 0x7faab81430c0 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.658+0000 7faab57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.658+0000 7faab57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.658+0000 7faab57fa640 10 monclient(hunting): dump: 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:11.658 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:11.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1784606783 0 0) 0x7faab8138a70 con 0x7faab80a98f0 2024-03-25T19:51:11.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 >> v1:172.21.15.18:6790/0 conn(0x7faab813f710 legacy=0x7faab8132c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 >> v1:172.21.15.53:6789/0 conn(0x7faab81430c0 legacy=0x7faab8133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:11.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 1 monclient: found mon.a 2024-03-25T19:51:11.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:11.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:11.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab81366f0 con 0x7faab80a98f0 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faab57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 5 monclient: authenticate success, global_id 4683 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 10 monclient: _renew_subs 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab81369a0 con 0x7faab80a98f0 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 10 monclient: _renew_subs 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faab81dabe0 con 0x7faab80a98f0 2024-03-25T19:51:11.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.663+0000 7faabed82640 1 librados: init done 2024-03-25T19:51:11.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.664+0000 7faabed82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.664+0000 7faabed82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.664+0000 7faabed82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:11.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:11.664+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab80b3ef0 con 0x7faab80a98f0 2024-03-25T19:51:12.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.007+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaac0035f0 con 0x7faab80a98f0 2024-03-25T19:51:12.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.007+0000 7faab57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:12.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.007+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7faaac001d80 con 0x7faab80a98f0 2024-03-25T19:51:12.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.007+0000 7faab57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:12.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.007+0000 7faab57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:12.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.007+0000 7faab57fa640 10 monclient: dump: 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:12.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.007+0000 7faab6ffd640 4 set_mon_vals no callback set 2024-03-25T19:51:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.008+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7faaac0197f0 con 0x7faab80a98f0 2024-03-25T19:51:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.008+0000 7faab7fff640 10 monclient: get_auth_request con 0x7faa9c06df70 auth_method 0 2024-03-25T19:51:12.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.008+0000 7faab57fa640 10 client.4683.objecter ms_handle_connect 0x7faa9c06df70 2024-03-25T19:51:12.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.013+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7faaac053510 con 0x7faab80a98f0 2024-03-25T19:51:12.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.013+0000 7faab57fa640 10 client.4683.objecter ms_dispatch 0x7faab8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:12.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.013+0000 7faab57fa640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:12.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.013+0000 7faab57fa640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:12.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.013+0000 7faab57fa640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T19:51:12.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.014+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faaac04ee50 con 0x7faab80a98f0 2024-03-25T19:51:12.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.014+0000 7faab57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:12.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.014+0000 7faab57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:12.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.230+0000 7faabed82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.230+0000 7faabed82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.230+0000 7faabed82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:12.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.230+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faab80627b0 con 0x7faab80a98f0 2024-03-25T19:51:12.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.312+0000 7faab57fa640 1 -- 172.21.15.18:0/2572666147 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7faaac050f50 con 0x7faab80a98f0 2024-03-25T19:51:12.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.312+0000 7faab57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.312+0000 7faab57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:12.312 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:51:12.312 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:12.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.314+0000 7faabed82640 10 librados: watch_flush enter 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.314+0000 7faabed82640 10 librados: watch_flush exit 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.314+0000 7faabed82640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.314+0000 7faabed82640 10 client.4683.objecter successfully canceled tick 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 >> v1:172.21.15.18:6800/1520000573 conn(0x7faa9c06df70 legacy=0x7faa9c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 10 monclient: shutdown 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 >> v1:172.21.15.18:6789/0 conn(0x7faab80a98f0 legacy=0x7faab81372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 shutdown_connections 2024-03-25T19:51:12.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 >> 172.21.15.18:0/2572666147 conn(0x7faab80b7ea0 msgr2=0x7faab8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:12.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 shutdown_connections 2024-03-25T19:51:12.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 1 -- 172.21.15.18:0/2572666147 wait complete. 2024-03-25T19:51:12.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:12.315+0000 7faabed82640 1 librados: shutdown 2024-03-25T19:51:12.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:19.067 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:46.597 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.072 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:46.603 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.112 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:46.642 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.135 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:19.140 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:46.670 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.142 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:46.638 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.148 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:46.679 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.152 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:46.648 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.177 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:10:46.673 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.182 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:46.713 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:19.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.240+0000 7f199e179640 10 monclient: get_monmap_and_config 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.240+0000 7f199e179640 10 monclient: build_initial_monmap 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.240+0000 7f199e179640 10 monclient: monmap: 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:51:19.241152+0000 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:51:19.241152+0000 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:51:19.240 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 1 Processor -- start 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 1 -- start start 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 10 monclient: init 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 10 monclient(hunting): picked mon.noname-b con 0x7f19980a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 10 monclient(hunting): picked mon.noname-a con 0x7f199813f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 10 monclient(hunting): picked mon.noname-c con 0x7f19981430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1998142520 con 0x7f199813f710 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19980b28a0 con 0x7f19981430c0 2024-03-25T19:51:19.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19980b2a30 con 0x7f19980a98f0 2024-03-25T19:51:19.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:19.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.241+0000 7f199e179640 10 monclient(hunting): authenticate will time out at 5506.368579s 2024-03-25T19:51:19.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.254+0000 7f1997fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f19981430c0 0x7f199814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:33328/0 (socket says 172.21.15.18:33328) 2024-03-25T19:51:19.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.254+0000 7f1997fff640 1 -- 172.21.15.18:0/479245129 learned_addr learned my addr 172.21.15.18:0/479245129 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:19.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.291+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903514218 0 0) 0x7f19980b28a0 con 0x7f19981430c0 2024-03-25T19:51:19.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.291+0000 7f19967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.291+0000 7f19967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.291+0000 7f19967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.291+0000 7f19967fc640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:51:19.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.291+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1980003520 con 0x7f19981430c0 2024-03-25T19:51:19.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1853843104 0 0) 0x7f1980003520 con 0x7f19981430c0 2024-03-25T19:51:19.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 >> v1:172.21.15.18:6790/0 conn(0x7f19980a98f0 legacy=0x7f19980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 >> v1:172.21.15.18:6789/0 conn(0x7f199813f710 legacy=0x7f19981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:19.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 1 monclient: found mon.noname-c 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19980b44c0 con 0x7f19981430c0 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1990001a10 con 0x7f19981430c0 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f199e179640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f199e179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f19967fc640 10 monclient: dump: 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:19.309 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:19.310 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:19.310 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.310 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:19.310 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:19.310 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:19.310 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:19.310 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:19.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.308+0000 7f199e179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:19.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.333+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1990001b70 con 0x7f19981430c0 2024-03-25T19:51:19.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.333+0000 7f19967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:19.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.333+0000 7f19967fc640 1 -- 172.21.15.18:0/479245129 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1990001fc0 con 0x7f19981430c0 2024-03-25T19:51:19.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.333+0000 7f19967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.333+0000 7f19967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:19.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.333+0000 7f19967fc640 10 monclient: dump: 2024-03-25T19:51:19.333 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:19.333 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.333+0000 7f199e179640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:19.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.333+0000 7f199e179640 4 set_mon_vals no callback set 2024-03-25T19:51:19.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.334+0000 7f199e179640 10 monclient: shutdown 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.334+0000 7f199e179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.334+0000 7f199e179640 1 -- 172.21.15.18:0/479245129 >> v1:172.21.15.53:6789/0 conn(0x7f19981430c0 legacy=0x7f199814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.334+0000 7f199e179640 1 -- 172.21.15.18:0/479245129 shutdown_connections 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.334+0000 7f199e179640 1 -- 172.21.15.18:0/479245129 >> 172.21.15.18:0/479245129 conn(0x7f19980b7ea0 msgr2=0x7f19980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.334+0000 7f199e179640 1 -- 172.21.15.18:0/479245129 shutdown_connections 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.334+0000 7f199e179640 1 -- 172.21.15.18:0/479245129 wait complete. 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.335+0000 7f199e179640 10 monclient: build_initial_monmap 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.335+0000 7f199e179640 10 monclient: monmap: 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:19.336 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.335+0000 7f199e179640 1 librados: starting msgr at 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.335+0000 7f199e179640 1 librados: starting objecter 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.335+0000 7f199e179640 1 Processor -- start 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 1 -- start start 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 1 librados: setting wanted keys 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 1 librados: calling monclient init 2024-03-25T19:51:19.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 10 monclient: init 2024-03-25T19:51:19.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:19.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:19.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 10 monclient(hunting): picked mon.noname-a con 0x7f19980a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:19.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 10 monclient(hunting): picked mon.noname-b con 0x7f199813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:19.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 10 monclient(hunting): picked mon.noname-c con 0x7f19981430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:19.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1998137ab0 con 0x7f19980a98f0 2024-03-25T19:51:19.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1998138bd0 con 0x7f199813f710 2024-03-25T19:51:19.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1998138e40 con 0x7f19981430c0 2024-03-25T19:51:19.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:19.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.336+0000 7f199e179640 10 monclient(hunting): authenticate will time out at 5506.463700s 2024-03-25T19:51:19.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.350+0000 7f1996ffd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f199813f710 0x7f19981df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:33354/0 (socket says 172.21.15.18:33354) 2024-03-25T19:51:19.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.350+0000 7f1996ffd640 1 -- 172.21.15.18:0/1234402239 learned_addr learned my addr 172.21.15.18:0/1234402239 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:19.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.353+0000 7f197ffff640 10 client.?.objecter ms_handle_connect 0x7f199813f710 2024-03-25T19:51:19.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.353+0000 7f197ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.360+0000 7f197ffff640 10 client.?.objecter ms_handle_connect 0x7f19981430c0 2024-03-25T19:51:19.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.360+0000 7f197ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.368+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248392916 0 0) 0x7f1998138e40 con 0x7f19981430c0 2024-03-25T19:51:19.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.368+0000 7f197ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.368+0000 7f197ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.368+0000 7f197ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.368+0000 7f197ffff640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:51:19.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.368+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1984004eb0 con 0x7f19981430c0 2024-03-25T19:51:19.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.370+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216434738 0 0) 0x7f1998138bd0 con 0x7f199813f710 2024-03-25T19:51:19.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.370+0000 7f197ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.370+0000 7f197ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.370+0000 7f197ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.370+0000 7f197ffff640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:51:19.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.370+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1998138e40 con 0x7f199813f710 2024-03-25T19:51:19.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.371+0000 7f197ffff640 10 client.?.objecter ms_handle_connect 0x7f19980a98f0 2024-03-25T19:51:19.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.371+0000 7f197ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.387+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3210752692 0 0) 0x7f1984004eb0 con 0x7f19981430c0 2024-03-25T19:51:19.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.387+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1998138bd0 con 0x7f19981430c0 2024-03-25T19:51:19.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.387+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1990001e80 con 0x7f19981430c0 2024-03-25T19:51:19.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.387+0000 7f197ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.387+0000 7f197ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:19.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.387+0000 7f197ffff640 10 monclient(hunting): dump: 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:19.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.387+0000 7f197ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1580758784 0 0) 0x7f1998138bd0 con 0x7f19981430c0 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 >> v1:172.21.15.53:6789/0 conn(0x7f199813f710 legacy=0x7f19981df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 >> v1:172.21.15.18:6789/0 conn(0x7f19980a98f0 legacy=0x7f1998137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 1 monclient: found mon.c 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1998139120 con 0x7f19981430c0 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 10 monclient: _finish_auth 0 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f197ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.396+0000 7f199e179640 5 monclient: authenticate success, global_id 4664 2024-03-25T19:51:19.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.397+0000 7f199e179640 10 monclient: _renew_subs 2024-03-25T19:51:19.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.397+0000 7f199e179640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:19.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.397+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1998139400 con 0x7f19981430c0 2024-03-25T19:51:19.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.397+0000 7f199e179640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:19.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.397+0000 7f199e179640 10 monclient: _renew_subs 2024-03-25T19:51:19.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.397+0000 7f199e179640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:19.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.397+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19980b7880 con 0x7f19981430c0 2024-03-25T19:51:19.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.397+0000 7f199e179640 1 librados: init done 2024-03-25T19:51:19.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.398+0000 7f199e179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.398+0000 7f199e179640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.398+0000 7f199e179640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.398+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1998137810 con 0x7f19981430c0 2024-03-25T19:51:19.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.710+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1990004e10 con 0x7f19981430c0 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.710+0000 7f197ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.710+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f1990002100 con 0x7f19981430c0 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.710+0000 7f197ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.710+0000 7f197ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.710+0000 7f197ffff640 10 monclient: dump: 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:19.711 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:19.712 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:19.712 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:19.712 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:19.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.710+0000 7f1995ffb640 4 set_mon_vals no callback set 2024-03-25T19:51:19.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.711+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f199001af90 con 0x7f19981430c0 2024-03-25T19:51:19.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.712+0000 7f19977fe640 10 monclient: get_auth_request con 0x7f198406dd50 auth_method 0 2024-03-25T19:51:19.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.712+0000 7f197ffff640 10 client.4664.objecter ms_handle_connect 0x7f198406dd50 2024-03-25T19:51:19.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.719+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7f1990054b50 con 0x7f19981430c0 2024-03-25T19:51:19.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.719+0000 7f197ffff640 10 client.4664.objecter ms_dispatch 0x7f1998000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:19.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.719+0000 7f197ffff640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:19.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.719+0000 7f197ffff640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:19.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.720+0000 7f197ffff640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-25T19:51:19.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.720+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1990050490 con 0x7f19981430c0 2024-03-25T19:51:19.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.720+0000 7f197ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.720+0000 7f197ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:19.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.986+0000 7f199e179640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.986+0000 7f199e179640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.986+0000 7f199e179640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:19.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:19.986+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19980627b0 con 0x7f19981430c0 2024-03-25T19:51:20.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.009+0000 7f197ffff640 1 -- 172.21.15.18:0/1234402239 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7f1990052590 con 0x7f19981430c0 2024-03-25T19:51:20.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.009+0000 7f197ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:20.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.009+0000 7f197ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:20.009 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:51:20.009 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:20.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.011+0000 7f199e179640 10 librados: watch_flush enter 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 10 librados: watch_flush exit 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 10 client.4664.objecter successfully canceled tick 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 >> v1:172.21.15.18:6800/1520000573 conn(0x7f198406dd50 legacy=0x7f1984070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 10 monclient: shutdown 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 >> v1:172.21.15.18:6790/0 conn(0x7f19981430c0 legacy=0x7f19981e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 shutdown_connections 2024-03-25T19:51:20.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 >> 172.21.15.18:0/1234402239 conn(0x7f19980b7ea0 msgr2=0x7f1998074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:20.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 shutdown_connections 2024-03-25T19:51:20.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 1 -- 172.21.15.18:0/1234402239 wait complete. 2024-03-25T19:51:20.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:20.012+0000 7f199e179640 1 librados: shutdown 2024-03-25T19:51:20.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:26.831 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.943+0000 7ff830b27640 10 monclient: get_monmap_and_config 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.943+0000 7ff830b27640 10 monclient: build_initial_monmap 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.943+0000 7ff830b27640 10 monclient: monmap: 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:51:26.944473+0000 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:51:26.944473+0000 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:26.943 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:26.944 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:26.944 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:51:26.944 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:51:26.944 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.944+0000 7ff830b27640 1 Processor -- start 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.944+0000 7ff830b27640 1 -- start start 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.944+0000 7ff830b27640 10 monclient: init 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.944+0000 7ff830b27640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.944+0000 7ff830b27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.944+0000 7ff830b27640 10 monclient(hunting): picked mon.noname-b con 0x7ff82c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.944+0000 7ff830b27640 10 monclient(hunting): picked mon.noname-c con 0x7ff82c13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.944+0000 7ff830b27640 10 monclient(hunting): picked mon.noname-a con 0x7ff82c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:26.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.945+0000 7ff830b27640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c142520 con 0x7ff82c1430c0 2024-03-25T19:51:26.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.945+0000 7ff830b27640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c0b28a0 con 0x7ff82c13f710 2024-03-25T19:51:26.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.945+0000 7ff830b27640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c0b2a30 con 0x7ff82c0a98f0 2024-03-25T19:51:26.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.945+0000 7ff830b27640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:26.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.945+0000 7ff830b27640 10 monclient(hunting): authenticate will time out at 5514.071768s 2024-03-25T19:51:26.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.945+0000 7ff823fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff82c13f710 0x7ff82c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:35564/0 (socket says 172.21.15.18:35564) 2024-03-25T19:51:26.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:26.945+0000 7ff823fff640 1 -- 172.21.15.18:0/3234897083 learned_addr learned my addr 172.21.15.18:0/3234897083 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:27.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.019+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746509821 0 0) 0x7ff82c0b28a0 con 0x7ff82c13f710 2024-03-25T19:51:27.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.020+0000 7ff829d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:51:27.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.020+0000 7ff829d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:27.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.020+0000 7ff829d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:27.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.020+0000 7ff829d9b640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:51:27.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.020+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff818003520 con 0x7ff82c13f710 2024-03-25T19:51:27.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.026+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563709757 0 0) 0x7ff82c142520 con 0x7ff82c1430c0 2024-03-25T19:51:27.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.026+0000 7ff829d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:51:27.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.026+0000 7ff829d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:27.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.026+0000 7ff829d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:27.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.026+0000 7ff829d9b640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:51:27.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.026+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c0b28a0 con 0x7ff82c1430c0 2024-03-25T19:51:27.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.029+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514967938 0 0) 0x7ff82c0b2a30 con 0x7ff82c0a98f0 2024-03-25T19:51:27.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.029+0000 7ff829d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:51:27.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.029+0000 7ff829d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:27.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.029+0000 7ff829d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:27.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.029+0000 7ff829d9b640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:51:27.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.029+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c142520 con 0x7ff82c0a98f0 2024-03-25T19:51:27.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3382676203 0 0) 0x7ff818003520 con 0x7ff82c13f710 2024-03-25T19:51:27.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 >> v1:172.21.15.18:6790/0 conn(0x7ff82c0a98f0 legacy=0x7ff82c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 >> v1:172.21.15.18:6789/0 conn(0x7ff82c1430c0 legacy=0x7ff82c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:51:27.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 1 monclient: found mon.noname-c 2024-03-25T19:51:27.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c0b44c0 con 0x7ff82c13f710 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 10 monclient: _finish_auth 0 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff814001690 con 0x7ff82c13f710 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff829d9b640 10 monclient: dump: 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:27.039 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:27.040 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:27.040 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:27.040 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:27.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff830b27640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:51:27.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.038+0000 7ff830b27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:27.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.052+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8140017f0 con 0x7ff82c13f710 2024-03-25T19:51:27.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.052+0000 7ff829d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:27.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.052+0000 7ff829d9b640 1 -- 172.21.15.18:0/3234897083 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff814001c00 con 0x7ff82c13f710 2024-03-25T19:51:27.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.052+0000 7ff830b27640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:27.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.052+0000 7ff830b27640 4 set_mon_vals no callback set 2024-03-25T19:51:27.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.052+0000 7ff829d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:27.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.052+0000 7ff829d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.052+0000 7ff829d9b640 10 monclient: dump: 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:27.053 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.053+0000 7ff830b27640 10 monclient: shutdown 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.053+0000 7ff830b27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.053+0000 7ff830b27640 1 -- 172.21.15.18:0/3234897083 >> v1:172.21.15.53:6789/0 conn(0x7ff82c13f710 legacy=0x7ff82c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.053+0000 7ff830b27640 1 -- 172.21.15.18:0/3234897083 shutdown_connections 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.053+0000 7ff830b27640 1 -- 172.21.15.18:0/3234897083 >> 172.21.15.18:0/3234897083 conn(0x7ff82c0b7ea0 msgr2=0x7ff82c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.053+0000 7ff830b27640 1 -- 172.21.15.18:0/3234897083 shutdown_connections 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.053+0000 7ff830b27640 1 -- 172.21.15.18:0/3234897083 wait complete. 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.054+0000 7ff830b27640 10 monclient: build_initial_monmap 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.054+0000 7ff830b27640 10 monclient: monmap: 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:51:27.055 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.055+0000 7ff830b27640 1 librados: starting msgr at 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.055+0000 7ff830b27640 1 librados: starting objecter 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.055+0000 7ff830b27640 1 Processor -- start 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.055+0000 7ff830b27640 1 -- start start 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.055+0000 7ff830b27640 1 librados: setting wanted keys 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.055+0000 7ff830b27640 1 librados: calling monclient init 2024-03-25T19:51:27.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.055+0000 7ff830b27640 10 monclient: init 2024-03-25T19:51:27.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:27.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:27.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 10 monclient(hunting): picked mon.noname-b con 0x7ff82c0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:27.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 10 monclient(hunting): picked mon.noname-c con 0x7ff82c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:27.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 10 monclient(hunting): picked mon.noname-a con 0x7ff82c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:27.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c139a40 con 0x7ff82c1430c0 2024-03-25T19:51:27.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c133f20 con 0x7ff82c0a98f0 2024-03-25T19:51:27.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c0ab950 con 0x7ff82c13f710 2024-03-25T19:51:27.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:27.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.056+0000 7ff830b27640 10 monclient(hunting): authenticate will time out at 5514.183289s 2024-03-25T19:51:27.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.059+0000 7ff82a59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff82c0a98f0 0x7ff82c138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:35604/0 (socket says 172.21.15.18:35604) 2024-03-25T19:51:27.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.059+0000 7ff82a59c640 1 -- 172.21.15.18:0/765211998 learned_addr learned my addr 172.21.15.18:0/765211998 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:27.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.060+0000 7ff822ffd640 10 client.?.objecter ms_handle_connect 0x7ff82c0a98f0 2024-03-25T19:51:27.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.060+0000 7ff822ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:27.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.064+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124223114 0 0) 0x7ff82c133f20 con 0x7ff82c0a98f0 2024-03-25T19:51:27.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.064+0000 7ff822ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:27.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.064+0000 7ff822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:27.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.064+0000 7ff822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:27.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.064+0000 7ff822ffd640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:51:27.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.064+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80c004eb0 con 0x7ff82c0a98f0 2024-03-25T19:51:27.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.071+0000 7ff822ffd640 10 client.?.objecter ms_handle_connect 0x7ff82c13f710 2024-03-25T19:51:27.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.071+0000 7ff822ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:27.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.073+0000 7ff822ffd640 10 client.?.objecter ms_handle_connect 0x7ff82c1430c0 2024-03-25T19:51:27.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.073+0000 7ff822ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:27.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.078+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114670025 0 0) 0x7ff82c0ab950 con 0x7ff82c13f710 2024-03-25T19:51:27.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.078+0000 7ff822ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:27.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.078+0000 7ff822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:27.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.078+0000 7ff822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:27.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.078+0000 7ff822ffd640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:51:27.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.078+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c133f20 con 0x7ff82c13f710 2024-03-25T19:51:27.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.094+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511040946 0 0) 0x7ff82c139a40 con 0x7ff82c1430c0 2024-03-25T19:51:27.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.094+0000 7ff822ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:27.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.094+0000 7ff822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:27.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.094+0000 7ff822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:27.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.094+0000 7ff822ffd640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:51:27.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.094+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c0ab950 con 0x7ff82c1430c0 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.097+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928529849 0 0) 0x7ff80c004eb0 con 0x7ff82c0a98f0 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.097+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff82c139a40 con 0x7ff82c0a98f0 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.097+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff81c001a10 con 0x7ff82c0a98f0 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.097+0000 7ff822ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.097+0000 7ff822ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.097+0000 7ff822ffd640 10 monclient(hunting): dump: 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.097 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:27.098 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:27.098 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:27.098 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:27.098 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:27.098 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:27.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.097+0000 7ff822ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:27.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.099+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987862132 0 0) 0x7ff82c133f20 con 0x7ff82c13f710 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.099+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff80c004eb0 con 0x7ff82c13f710 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.099+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff814001b00 con 0x7ff82c13f710 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.099+0000 7ff822ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.099+0000 7ff822ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.099+0000 7ff822ffd640 10 monclient(hunting): dump: 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:27.099 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:27.100 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.103+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894602301 0 0) 0x7ff82c0ab950 con 0x7ff82c1430c0 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.103+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff82c133f20 con 0x7ff82c1430c0 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.103+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff824001bd0 con 0x7ff82c1430c0 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.103+0000 7ff822ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.103+0000 7ff822ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.103+0000 7ff822ffd640 10 monclient(hunting): dump: 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:27.103 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:27.104 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:27.104 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:27.104 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2364366045 0 0) 0x7ff82c139a40 con 0x7ff82c0a98f0 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 >> v1:172.21.15.18:6790/0 conn(0x7ff82c13f710 legacy=0x7ff82c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 >> v1:172.21.15.18:6789/0 conn(0x7ff82c1430c0 legacy=0x7ff82c133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 1 monclient: found mon.b 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c0abc30 con 0x7ff82c0a98f0 2024-03-25T19:51:27.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 5 monclient: authenticate success, global_id 4672 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 10 monclient: _renew_subs 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82c0abf10 con 0x7ff82c0a98f0 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 10 monclient: _renew_subs 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c139370 con 0x7ff82c0a98f0 2024-03-25T19:51:27.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.109+0000 7ff830b27640 1 librados: init done 2024-03-25T19:51:27.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.110+0000 7ff830b27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:27.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.110+0000 7ff830b27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:27.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.110+0000 7ff830b27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:27.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.110+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82c0627b0 con 0x7ff82c0a98f0 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.418+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff81c001c70 con 0x7ff82c0a98f0 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.418+0000 7ff822ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.418+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff81c003a60 con 0x7ff82c0a98f0 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.418+0000 7ff822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.418+0000 7ff822ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.418+0000 7ff822ffd640 10 monclient: dump: 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:27.418 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:27.419 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:27.419 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:27.419 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:27.419 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:27.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.418+0000 7ff82959a640 4 set_mon_vals no callback set 2024-03-25T19:51:27.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.419+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ff81c003c80 con 0x7ff82c0a98f0 2024-03-25T19:51:27.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.419+0000 7ff823fff640 10 monclient: get_auth_request con 0x7ff80c06e0f0 auth_method 0 2024-03-25T19:51:27.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.420+0000 7ff822ffd640 10 client.4672.objecter ms_handle_connect 0x7ff80c06e0f0 2024-03-25T19:51:27.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.425+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7ff81c054860 con 0x7ff82c0a98f0 2024-03-25T19:51:27.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.425+0000 7ff822ffd640 10 client.4672.objecter ms_dispatch 0x7ff82c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:27.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.425+0000 7ff822ffd640 3 client.4672.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:27.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.425+0000 7ff822ffd640 3 client.4672.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:27.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.425+0000 7ff822ffd640 20 client.4672.objecter dump_active .. 0 homeless 2024-03-25T19:51:27.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.425+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff81c050370 con 0x7ff82c0a98f0 2024-03-25T19:51:27.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.425+0000 7ff822ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:27.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.425+0000 7ff822ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:27.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.678+0000 7ff830b27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.678+0000 7ff830b27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.678+0000 7ff830b27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:27.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.678+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff82c0b7180 con 0x7ff82c0a98f0 2024-03-25T19:51:27.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.709+0000 7ff822ffd640 1 -- 172.21.15.18:0/765211998 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7ff81c0522a0 con 0x7ff82c0a98f0 2024-03-25T19:51:27.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.709+0000 7ff822ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.709+0000 7ff822ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:27.709 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:51:27.709 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:27.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 10 librados: watch_flush enter 2024-03-25T19:51:27.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 10 librados: watch_flush exit 2024-03-25T19:51:27.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 20 client.4672.objecter shutdown clearing up homeless session... 2024-03-25T19:51:27.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 10 client.4672.objecter successfully canceled tick 2024-03-25T19:51:27.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 >> v1:172.21.15.18:6800/1520000573 conn(0x7ff80c06e0f0 legacy=0x7ff80c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 10 monclient: shutdown 2024-03-25T19:51:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 >> v1:172.21.15.53:6789/0 conn(0x7ff82c0a98f0 legacy=0x7ff82c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 shutdown_connections 2024-03-25T19:51:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 >> 172.21.15.18:0/765211998 conn(0x7ff82c0b7ea0 msgr2=0x7ff82c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 shutdown_connections 2024-03-25T19:51:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 1 -- 172.21.15.18:0/765211998 wait complete. 2024-03-25T19:51:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:27.711+0000 7ff830b27640 1 librados: shutdown 2024-03-25T19:51:27.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:34.529 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:34.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.647+0000 7f99029cd640 10 monclient: get_monmap_and_config 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.647+0000 7f99029cd640 10 monclient: build_initial_monmap 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.648+0000 7f99029cd640 10 monclient: monmap: 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:51:34.649837+0000 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:51:34.649837+0000 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.648+0000 7f99029cd640 1 Processor -- start 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.648+0000 7f99029cd640 1 -- start start 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.648+0000 7f99029cd640 10 monclient: init 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 10 monclient(hunting): picked mon.noname-c con 0x7f98fc0a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 10 monclient(hunting): picked mon.noname-b con 0x7f98fc13f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 10 monclient(hunting): picked mon.noname-a con 0x7f98fc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc142520 con 0x7f98fc1430c0 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc0b28a0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc0b2a30 con 0x7f98fc13f710 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:34.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.649+0000 7f99029cd640 10 monclient(hunting): authenticate will time out at 5521.777420s 2024-03-25T19:51:34.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.654+0000 7f98fb7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f98fc13f710 0x7f98fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:42378/0 (socket says 172.21.15.18:42378) 2024-03-25T19:51:34.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.654+0000 7f98fb7fe640 1 -- 172.21.15.18:0/1689460820 learned_addr learned my addr 172.21.15.18:0/1689460820 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:34.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454640591 0 0) 0x7f98fc0b28a0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:51:34.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e4003520 con 0x7f98fc0a98f0 2024-03-25T19:51:34.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633971841 0 0) 0x7f98fc0b2a30 con 0x7f98fc13f710 2024-03-25T19:51:34.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:51:34.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.673+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98fc0b28a0 con 0x7f98fc13f710 2024-03-25T19:51:34.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.684+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122574488 0 0) 0x7f98fc142520 con 0x7f98fc1430c0 2024-03-25T19:51:34.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.684+0000 7f98faffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.684+0000 7f98faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.684+0000 7f98faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.684+0000 7f98faffd640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:51:34.685 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.684+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98fc0b2a30 con 0x7f98fc1430c0 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.693+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787906208 0 0) 0x7f98e4003520 con 0x7f98fc0a98f0 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.693+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 >> v1:172.21.15.18:6790/0 conn(0x7f98fc13f710 legacy=0x7f98fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 >> v1:172.21.15.18:6789/0 conn(0x7f98fc1430c0 legacy=0x7f98fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 1 monclient: found mon.noname-c 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0b44c0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f98ec001690 con 0x7f98fc0a98f0 2024-03-25T19:51:34.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f99029cd640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f99029cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f98faffd640 10 monclient: dump: 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:34.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.694+0000 7f99029cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:34.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.718+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98ec0017f0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.718+0000 7f98faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:34.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.718+0000 7f98faffd640 1 -- 172.21.15.18:0/1689460820 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f98ec001c00 con 0x7f98fc0a98f0 2024-03-25T19:51:34.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.718+0000 7f99029cd640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.718+0000 7f99029cd640 4 set_mon_vals no callback set 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.718+0000 7f98faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.718+0000 7f98faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.718+0000 7f98faffd640 10 monclient: dump: 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:34.720 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:34.721 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:34.721 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:34.721 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.719+0000 7f99029cd640 10 monclient: shutdown 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.719+0000 7f99029cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.719+0000 7f99029cd640 1 -- 172.21.15.18:0/1689460820 >> v1:172.21.15.53:6789/0 conn(0x7f98fc0a98f0 legacy=0x7f98fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.719+0000 7f99029cd640 1 -- 172.21.15.18:0/1689460820 shutdown_connections 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.719+0000 7f99029cd640 1 -- 172.21.15.18:0/1689460820 >> 172.21.15.18:0/1689460820 conn(0x7f98fc0b7ea0 msgr2=0x7f98fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.719+0000 7f99029cd640 1 -- 172.21.15.18:0/1689460820 shutdown_connections 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.719+0000 7f99029cd640 1 -- 172.21.15.18:0/1689460820 wait complete. 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.720+0000 7f99029cd640 10 monclient: build_initial_monmap 2024-03-25T19:51:34.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.720+0000 7f99029cd640 10 monclient: monmap: 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 librados: starting msgr at 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 librados: starting objecter 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 Processor -- start 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 -- start start 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 librados: setting wanted keys 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 librados: calling monclient init 2024-03-25T19:51:34.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 10 monclient: init 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 10 monclient(hunting): picked mon.noname-c con 0x7f98fc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 10 monclient(hunting): picked mon.noname-b con 0x7f98fc13f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 10 monclient(hunting): picked mon.noname-a con 0x7f98fc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc138250 con 0x7f98fc1430c0 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc137d50 con 0x7f98fc13f710 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc137ee0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:34.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.721+0000 7f99029cd640 10 monclient(hunting): authenticate will time out at 5521.849701s 2024-03-25T19:51:34.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.734+0000 7f98fbfff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f98fc0a98f0 0x7f98fc137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:42438/0 (socket says 172.21.15.18:42438) 2024-03-25T19:51:34.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.734+0000 7f98fbfff640 1 -- 172.21.15.18:0/2553779784 learned_addr learned my addr 172.21.15.18:0/2553779784 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:34.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.735+0000 7f98f8ff9640 10 client.?.objecter ms_handle_connect 0x7f98fc0a98f0 2024-03-25T19:51:34.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.736+0000 7f98f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.742+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229596263 0 0) 0x7f98fc137ee0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.742+0000 7f98f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.742+0000 7f98f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.742+0000 7f98f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.742+0000 7f98f8ff9640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:51:34.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.742+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e8004eb0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.749+0000 7f98f8ff9640 10 client.?.objecter ms_handle_connect 0x7f98fc13f710 2024-03-25T19:51:34.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.749+0000 7f98f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.754+0000 7f98f8ff9640 10 client.?.objecter ms_handle_connect 0x7f98fc1430c0 2024-03-25T19:51:34.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.754+0000 7f98f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.756+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1395613163 0 0) 0x7f98e8004eb0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.756+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98fc137ee0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.756+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f98ec001690 con 0x7f98fc0a98f0 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.756+0000 7f98f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.756+0000 7f98f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.756+0000 7f98f8ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:34.757 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:34.758 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.758 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:34.758 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:34.758 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:34.758 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:34.758 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:34.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.756+0000 7f98f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.763+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101566627 0 0) 0x7f98fc137d50 con 0x7f98fc13f710 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.763+0000 7f98f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.763+0000 7f98f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.763+0000 7f98f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.763+0000 7f98f8ff9640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.763+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e8004eb0 con 0x7f98fc13f710 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 93292528 0 0) 0x7f98fc137ee0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 >> v1:172.21.15.53:6789/0 conn(0x7f98fc13f710 legacy=0x7f98fc1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 >> v1:172.21.15.18:6789/0 conn(0x7f98fc1430c0 legacy=0x7f98fc1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 1 monclient: found mon.c 2024-03-25T19:51:34.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0b7730 con 0x7f98fc0a98f0 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f98f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 5 monclient: authenticate success, global_id 4676 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 10 monclient: _renew_subs 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc0b7920 con 0x7f98fc0a98f0 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 10 monclient: _renew_subs 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc1e33d0 con 0x7f98fc0a98f0 2024-03-25T19:51:34.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.764+0000 7f99029cd640 1 librados: init done 2024-03-25T19:51:34.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.765+0000 7f99029cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.765+0000 7f99029cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.765+0000 7f99029cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:34.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:34.765+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc0627b0 con 0x7f98fc0a98f0 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.076+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98ec0038a0 con 0x7f98fc0a98f0 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.076+0000 7f98f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.076+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f98ec003c70 con 0x7f98fc0a98f0 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.076+0000 7f98f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.076+0000 7f98f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.076+0000 7f98f8ff9640 10 monclient: dump: 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:35.077 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:35.078 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:35.078 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:35.078 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:35.078 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:35.078 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:35.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.076+0000 7f98fa7fc640 4 set_mon_vals no callback set 2024-03-25T19:51:35.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.077+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f98ec003e90 con 0x7f98fc0a98f0 2024-03-25T19:51:35.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.077+0000 7f98fb7fe640 10 monclient: get_auth_request con 0x7f98e806dd90 auth_method 0 2024-03-25T19:51:35.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.077+0000 7f98f8ff9640 10 client.4676.objecter ms_handle_connect 0x7f98e806dd90 2024-03-25T19:51:35.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.084+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7f98ec054900 con 0x7f98fc0a98f0 2024-03-25T19:51:35.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.084+0000 7f98f8ff9640 10 client.4676.objecter ms_dispatch 0x7f98fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:35.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.084+0000 7f98f8ff9640 3 client.4676.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:35.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.084+0000 7f98f8ff9640 3 client.4676.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:35.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.084+0000 7f98f8ff9640 20 client.4676.objecter dump_active .. 0 homeless 2024-03-25T19:51:35.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.084+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98ec050410 con 0x7f98fc0a98f0 2024-03-25T19:51:35.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.084+0000 7f98f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:35.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.084+0000 7f98f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.340+0000 7f99029cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.340+0000 7f99029cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.340+0000 7f99029cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:35.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.340+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98fc13df50 con 0x7f98fc0a98f0 2024-03-25T19:51:35.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.372+0000 7f98f8ff9640 1 -- 172.21.15.18:0/2553779784 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7f98ec052340 con 0x7f98fc0a98f0 2024-03-25T19:51:35.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.372+0000 7f98f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.372+0000 7f98f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:35.374 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:51:35.374 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:35.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 10 librados: watch_flush enter 2024-03-25T19:51:35.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 10 librados: watch_flush exit 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 20 client.4676.objecter shutdown clearing up homeless session... 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 10 client.4676.objecter successfully canceled tick 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 >> v1:172.21.15.18:6800/1520000573 conn(0x7f98e806dd90 legacy=0x7f98e8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 10 monclient: shutdown 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 >> v1:172.21.15.18:6790/0 conn(0x7f98fc0a98f0 legacy=0x7f98fc137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.374+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 shutdown_connections 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.375+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 >> 172.21.15.18:0/2553779784 conn(0x7f98fc0b7ea0 msgr2=0x7f98fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.375+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 shutdown_connections 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.375+0000 7f99029cd640 1 -- 172.21.15.18:0/2553779784 wait complete. 2024-03-25T19:51:35.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:35.375+0000 7f99029cd640 1 librados: shutdown 2024-03-25T19:51:35.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:42.194 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.310+0000 7ff8567ed640 10 monclient: get_monmap_and_config 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.310+0000 7ff8567ed640 10 monclient: build_initial_monmap 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.310+0000 7ff8567ed640 10 monclient: monmap: 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:51:42.311880+0000 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:51:42.311880+0000 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.310+0000 7ff8567ed640 1 Processor -- start 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.310+0000 7ff8567ed640 1 -- start start 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.310+0000 7ff8567ed640 10 monclient: init 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 10 monclient(hunting): picked mon.noname-c con 0x7ff8500705b0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 10 monclient(hunting): picked mon.noname-a con 0x7ff8501402d0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 10 monclient(hunting): picked mon.noname-b con 0x7ff850149ae0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8500718c0 con 0x7ff8501402d0 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8500714c0 con 0x7ff8500705b0 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff850072b50 con 0x7ff850149ae0 2024-03-25T19:51:42.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:42.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.311+0000 7ff8567ed640 10 monclient(hunting): authenticate will time out at 5529.439688s 2024-03-25T19:51:42.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.313+0000 7ff84ffff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff8500705b0 0x7ff85013d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:47544/0 (socket says 172.21.15.18:47544) 2024-03-25T19:51:42.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.313+0000 7ff84ffff640 1 -- 172.21.15.18:0/478137792 learned_addr learned my addr 172.21.15.18:0/478137792 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:42.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.329+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653388349 0 0) 0x7ff8500714c0 con 0x7ff8500705b0 2024-03-25T19:51:42.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.329+0000 7ff84effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.329+0000 7ff84effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.329+0000 7ff84effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.329+0000 7ff84effd640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:51:42.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.329+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff838003520 con 0x7ff8500705b0 2024-03-25T19:51:42.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.331+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256843350 0 0) 0x7ff850072b50 con 0x7ff850149ae0 2024-03-25T19:51:42.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.331+0000 7ff84effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.331+0000 7ff84effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.331+0000 7ff84effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.331+0000 7ff84effd640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:51:42.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.331+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8500714c0 con 0x7ff850149ae0 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.336+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878420118 0 0) 0x7ff8500718c0 con 0x7ff8501402d0 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.336+0000 7ff84effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.336+0000 7ff84effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.336+0000 7ff84effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.336+0000 7ff84effd640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.336+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff850072b50 con 0x7ff8501402d0 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.338+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3932896067 0 0) 0x7ff838003520 con 0x7ff8500705b0 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 >> v1:172.21.15.18:6790/0 conn(0x7ff850149ae0 legacy=0x7ff85014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 >> v1:172.21.15.18:6789/0 conn(0x7ff8501402d0 legacy=0x7ff850140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 1 monclient: found mon.noname-c 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85013dd10 con 0x7ff8500705b0 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 10 monclient: _finish_auth 0 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff840001690 con 0x7ff8500705b0 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff8567ed640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:51:42.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff8567ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff84effd640 10 monclient: dump: 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:42.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.339+0000 7ff8567ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:42.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.347+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8400017f0 con 0x7ff8500705b0 2024-03-25T19:51:42.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.347+0000 7ff84effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:42.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.347+0000 7ff84effd640 1 -- 172.21.15.18:0/478137792 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff840001c00 con 0x7ff8500705b0 2024-03-25T19:51:42.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.347+0000 7ff84effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.347+0000 7ff84effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.347+0000 7ff84effd640 10 monclient: dump: 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:42.350 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:42.350 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:42.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.348+0000 7ff8567ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:42.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.348+0000 7ff8567ed640 4 set_mon_vals no callback set 2024-03-25T19:51:42.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.348+0000 7ff8567ed640 10 monclient: shutdown 2024-03-25T19:51:42.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.348+0000 7ff8567ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.348+0000 7ff8567ed640 1 -- 172.21.15.18:0/478137792 >> v1:172.21.15.53:6789/0 conn(0x7ff8500705b0 legacy=0x7ff85013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.349+0000 7ff8567ed640 1 -- 172.21.15.18:0/478137792 shutdown_connections 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.349+0000 7ff8567ed640 1 -- 172.21.15.18:0/478137792 >> 172.21.15.18:0/478137792 conn(0x7ff8500b5d10 msgr2=0x7ff8500b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.349+0000 7ff8567ed640 1 -- 172.21.15.18:0/478137792 shutdown_connections 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.349+0000 7ff8567ed640 1 -- 172.21.15.18:0/478137792 wait complete. 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.350+0000 7ff8567ed640 10 monclient: build_initial_monmap 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.350+0000 7ff8567ed640 10 monclient: monmap: 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:51:42.352 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.350+0000 7ff8567ed640 1 librados: starting msgr at 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.350+0000 7ff8567ed640 1 librados: starting objecter 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.350+0000 7ff8567ed640 1 Processor -- start 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.351+0000 7ff8567ed640 1 -- start start 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.351+0000 7ff8567ed640 1 librados: setting wanted keys 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.351+0000 7ff8567ed640 1 librados: calling monclient init 2024-03-25T19:51:42.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.351+0000 7ff8567ed640 10 monclient: init 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.351+0000 7ff8567ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.351+0000 7ff8567ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.351+0000 7ff8567ed640 10 monclient(hunting): picked mon.noname-a con 0x7ff8500705b0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.352+0000 7ff8567ed640 10 monclient(hunting): picked mon.noname-c con 0x7ff8501402d0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.352+0000 7ff8567ed640 10 monclient(hunting): picked mon.noname-b con 0x7ff850149ae0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.352+0000 7ff8567ed640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85006d690 con 0x7ff8500705b0 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.352+0000 7ff8567ed640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff850134870 con 0x7ff850149ae0 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.352+0000 7ff8567ed640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85006c420 con 0x7ff8501402d0 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.352+0000 7ff8567ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:42.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.352+0000 7ff8567ed640 10 monclient(hunting): authenticate will time out at 5529.480029s 2024-03-25T19:51:42.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.360+0000 7ff84ffff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7ff8500705b0 0x7ff8501dab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41696/0 (socket says 172.21.15.18:41696) 2024-03-25T19:51:42.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.360+0000 7ff84ffff640 1 -- 172.21.15.18:0/1860947347 learned_addr learned my addr 172.21.15.18:0/1860947347 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:42.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.365+0000 7ff84cff9640 10 client.?.objecter ms_handle_connect 0x7ff850149ae0 2024-03-25T19:51:42.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.365+0000 7ff84cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.367+0000 7ff84cff9640 10 client.?.objecter ms_handle_connect 0x7ff8500705b0 2024-03-25T19:51:42.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.367+0000 7ff84cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.370+0000 7ff84cff9640 10 client.?.objecter ms_handle_connect 0x7ff8501402d0 2024-03-25T19:51:42.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.370+0000 7ff84cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.373+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132813158 0 0) 0x7ff85006d690 con 0x7ff8500705b0 2024-03-25T19:51:42.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.373+0000 7ff84cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.373+0000 7ff84cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.373+0000 7ff84cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.373+0000 7ff84cff9640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:51:42.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.373+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c004eb0 con 0x7ff8500705b0 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.376+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970908730 0 0) 0x7ff850134870 con 0x7ff850149ae0 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.376+0000 7ff84cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.376+0000 7ff84cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.376+0000 7ff84cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.376+0000 7ff84cff9640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.376+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff85006d690 con 0x7ff850149ae0 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.384+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2286045594 0 0) 0x7ff83c004eb0 con 0x7ff8500705b0 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.384+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff850134870 con 0x7ff8500705b0 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.384+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff840001bc0 con 0x7ff8500705b0 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.384+0000 7ff84cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.384+0000 7ff84cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.384+0000 7ff84cff9640 10 monclient(hunting): dump: 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.384+0000 7ff84cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:42.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.388+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066285565 0 0) 0x7ff85006c420 con 0x7ff8501402d0 2024-03-25T19:51:42.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.389+0000 7ff84cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.389+0000 7ff84cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.389+0000 7ff84cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.389+0000 7ff84cff9640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:51:42.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.389+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83c004eb0 con 0x7ff8501402d0 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1386436662 0 0) 0x7ff850134870 con 0x7ff8500705b0 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 >> v1:172.21.15.18:6790/0 conn(0x7ff8501402d0 legacy=0x7ff850133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 >> v1:172.21.15.53:6789/0 conn(0x7ff850149ae0 legacy=0x7ff850134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 1 monclient: found mon.a 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85006c0b0 con 0x7ff8500705b0 2024-03-25T19:51:42.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.396+0000 7ff8567ed640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.397+0000 7ff8567ed640 10 monclient: _renew_subs 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.397+0000 7ff8567ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.397+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85013fec0 con 0x7ff8500705b0 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.397+0000 7ff8567ed640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.397+0000 7ff8567ed640 10 monclient: _renew_subs 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.397+0000 7ff8567ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.397+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85006be70 con 0x7ff8500705b0 2024-03-25T19:51:42.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.397+0000 7ff8567ed640 1 librados: init done 2024-03-25T19:51:42.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.398+0000 7ff8567ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.398+0000 7ff8567ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.398+0000 7ff8567ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:42.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.398+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8500627b0 con 0x7ff8500705b0 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.704+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8400011d0 con 0x7ff8500705b0 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.704+0000 7ff84cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.704+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ff840004cb0 con 0x7ff8500705b0 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.704+0000 7ff84cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.704+0000 7ff84cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.704+0000 7ff84cff9640 10 monclient: dump: 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:42.706 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:42.707 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:42.707 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.707 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:42.707 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:42.707 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:42.707 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:42.707 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:42.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.704+0000 7ff84e7fc640 4 set_mon_vals no callback set 2024-03-25T19:51:42.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.706+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ff84001acc0 con 0x7ff8500705b0 2024-03-25T19:51:42.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.706+0000 7ff84f7fe640 10 monclient: get_auth_request con 0x7ff83c06dd60 auth_method 0 2024-03-25T19:51:42.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.706+0000 7ff84cff9640 10 client.4707.objecter ms_handle_connect 0x7ff83c06dd60 2024-03-25T19:51:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.713+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7ff8400549a0 con 0x7ff8500705b0 2024-03-25T19:51:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.713+0000 7ff84cff9640 10 client.4707.objecter ms_dispatch 0x7ff850000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.713+0000 7ff84cff9640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.713+0000 7ff84cff9640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.713+0000 7ff84cff9640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T19:51:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.713+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8400502e0 con 0x7ff8500705b0 2024-03-25T19:51:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.713+0000 7ff84cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.713+0000 7ff84cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:42.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.967+0000 7ff8567ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.967+0000 7ff8567ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.967+0000 7ff8567ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:51:42.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.967+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff850140050 con 0x7ff8500705b0 2024-03-25T19:51:42.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.993+0000 7ff84cff9640 1 -- 172.21.15.18:0/1860947347 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7ff8400523e0 con 0x7ff8500705b0 2024-03-25T19:51:42.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.993+0000 7ff84cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.993+0000 7ff84cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:42.994 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:51:42.995 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 10 librados: watch_flush enter 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 10 librados: watch_flush exit 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 10 client.4707.objecter successfully canceled tick 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 >> v1:172.21.15.18:6800/1520000573 conn(0x7ff83c06dd60 legacy=0x7ff83c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 10 monclient: shutdown 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 >> v1:172.21.15.18:6789/0 conn(0x7ff8500705b0 legacy=0x7ff8501dab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 shutdown_connections 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.995+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 >> 172.21.15.18:0/1860947347 conn(0x7ff8500b5d10 msgr2=0x7ff85013c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.996+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 shutdown_connections 2024-03-25T19:51:42.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.996+0000 7ff8567ed640 1 -- 172.21.15.18:0/1860947347 wait complete. 2024-03-25T19:51:42.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:42.996+0000 7ff8567ed640 1 librados: shutdown 2024-03-25T19:51:43.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:44.136 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:11.666 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.143 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:11.674 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.189 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:11.719 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.194 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:11.690 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.195 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:11.725 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.200 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:11.696 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.206 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:11.737 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.226 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:11.723 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:44.237 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:11.768 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:49.815 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:49.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.934+0000 7f5aa4e9a640 10 monclient: get_monmap_and_config 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.934+0000 7f5aa4e9a640 10 monclient: build_initial_monmap 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.934+0000 7f5aa4e9a640 10 monclient: monmap: 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:51:49.936046+0000 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:51:49.936046+0000 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:51:49.935 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:49.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.935+0000 7f5aa4e9a640 1 Processor -- start 2024-03-25T19:51:49.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.935+0000 7f5aa4e9a640 1 -- start start 2024-03-25T19:51:49.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.935+0000 7f5aa4e9a640 10 monclient: init 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f5aa00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f5aa013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f5aa01430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa0142520 con 0x7f5aa00a98f0 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa00b28a0 con 0x7f5aa01430c0 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa00b2a30 con 0x7f5aa013f710 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:49.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.936+0000 7f5aa4e9a640 10 monclient(hunting): authenticate will time out at 5537.064065s 2024-03-25T19:51:49.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.943+0000 7f5a9dd9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f5aa013f710 0x7f5aa01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:56056/0 (socket says 172.21.15.18:56056) 2024-03-25T19:51:49.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:49.943+0000 7f5a9dd9b640 1 -- 172.21.15.18:0/972806752 learned_addr learned my addr 172.21.15.18:0/972806752 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.036+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50033071 0 0) 0x7f5aa0142520 con 0x7f5aa00a98f0 2024-03-25T19:51:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.036+0000 7f5a9d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.036+0000 7f5a9d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.036+0000 7f5a9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.036+0000 7f5a9d59a640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:51:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.036+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a88003520 con 0x7f5aa00a98f0 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146675146 0 0) 0x7f5a88003520 con 0x7f5aa00a98f0 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 >> v1:172.21.15.18:6790/0 conn(0x7f5aa013f710 legacy=0x7f5aa01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 >> v1:172.21.15.53:6789/0 conn(0x7f5aa01430c0 legacy=0x7f5aa014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 1 monclient: found mon.noname-a 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa00b44c0 con 0x7f5aa00a98f0 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 10 monclient: _finish_auth 0 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5a90001690 con 0x7f5aa00a98f0 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5a9d59a640 10 monclient: dump: 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5aa4e9a640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:51:50.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.058+0000 7f5aa4e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:50.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.078+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a900017f0 con 0x7f5aa00a98f0 2024-03-25T19:51:50.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.078+0000 7f5a9d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:50.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.078+0000 7f5a9d59a640 1 -- 172.21.15.18:0/972806752 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5a90001c00 con 0x7f5aa00a98f0 2024-03-25T19:51:50.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.078+0000 7f5a9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.078+0000 7f5a9d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:50.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.078+0000 7f5a9d59a640 10 monclient: dump: 2024-03-25T19:51:50.079 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.078+0000 7f5aa4e9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.078+0000 7f5aa4e9a640 4 set_mon_vals no callback set 2024-03-25T19:51:50.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.079+0000 7f5aa4e9a640 10 monclient: shutdown 2024-03-25T19:51:50.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.079+0000 7f5aa4e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.079+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/972806752 >> v1:172.21.15.18:6789/0 conn(0x7f5aa00a98f0 legacy=0x7f5aa00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.079+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/972806752 shutdown_connections 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.079+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/972806752 >> 172.21.15.18:0/972806752 conn(0x7f5aa00b7ea0 msgr2=0x7f5aa00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.079+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/972806752 shutdown_connections 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.079+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/972806752 wait complete. 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.080+0000 7f5aa4e9a640 10 monclient: build_initial_monmap 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.080+0000 7f5aa4e9a640 10 monclient: monmap: 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:50.082 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.081+0000 7f5aa4e9a640 1 librados: starting msgr at 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.081+0000 7f5aa4e9a640 1 librados: starting objecter 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.081+0000 7f5aa4e9a640 1 Processor -- start 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.081+0000 7f5aa4e9a640 1 -- start start 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.081+0000 7f5aa4e9a640 1 librados: setting wanted keys 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.081+0000 7f5aa4e9a640 1 librados: calling monclient init 2024-03-25T19:51:50.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.081+0000 7f5aa4e9a640 10 monclient: init 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f5aa00a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f5aa013f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f5aa01430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa0135ac0 con 0x7f5aa013f710 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa0137030 con 0x7f5aa01430c0 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa01372a0 con 0x7f5aa00a98f0 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:50.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.082+0000 7f5aa4e9a640 10 monclient(hunting): authenticate will time out at 5537.210096s 2024-03-25T19:51:50.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.092+0000 7f5a9ed9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5aa01430c0 0x7f5aa01e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:56026/0 (socket says 172.21.15.18:56026) 2024-03-25T19:51:50.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.092+0000 7f5a9ed9d640 1 -- 172.21.15.18:0/1599514233 learned_addr learned my addr 172.21.15.18:0/1599514233 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:50.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.092+0000 7f5a8effd640 10 client.?.objecter ms_handle_connect 0x7f5aa01430c0 2024-03-25T19:51:50.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.092+0000 7f5a8effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:50.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.104+0000 7f5a8effd640 10 client.?.objecter ms_handle_connect 0x7f5aa013f710 2024-03-25T19:51:50.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.104+0000 7f5a8effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:50.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.111+0000 7f5a8effd640 10 client.?.objecter ms_handle_connect 0x7f5aa00a98f0 2024-03-25T19:51:50.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.111+0000 7f5a8effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:50.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.170+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027279322 0 0) 0x7f5aa0137030 con 0x7f5aa01430c0 2024-03-25T19:51:50.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.170+0000 7f5a8effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:50.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.170+0000 7f5a8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:50.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.170+0000 7f5a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:50.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.170+0000 7f5a8effd640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:51:50.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.170+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a80004eb0 con 0x7f5aa01430c0 2024-03-25T19:51:50.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.172+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972099646 0 0) 0x7f5aa0135ac0 con 0x7f5aa013f710 2024-03-25T19:51:50.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.172+0000 7f5a8effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:50.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.172+0000 7f5a8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:50.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.172+0000 7f5a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:50.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.172+0000 7f5a8effd640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:51:50.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.172+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa0137030 con 0x7f5aa013f710 2024-03-25T19:51:50.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.186+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 528607119 0 0) 0x7f5a80004eb0 con 0x7f5aa01430c0 2024-03-25T19:51:50.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.186+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aa0135ac0 con 0x7f5aa01430c0 2024-03-25T19:51:50.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.186+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5a98001c30 con 0x7f5aa01430c0 2024-03-25T19:51:50.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.186+0000 7f5a8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.186+0000 7f5a8effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:50.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.186+0000 7f5a8effd640 10 monclient(hunting): dump: 2024-03-25T19:51:50.187 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:50.187 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:50.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.186+0000 7f5a8effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:50.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.192+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464456153 0 0) 0x7f5aa01372a0 con 0x7f5aa00a98f0 2024-03-25T19:51:50.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.192+0000 7f5a8effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:50.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.192+0000 7f5a8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:50.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.192+0000 7f5a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:50.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.192+0000 7f5a8effd640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:51:50.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.192+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a80004eb0 con 0x7f5aa00a98f0 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.200+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303553455 0 0) 0x7f5aa0137030 con 0x7f5aa013f710 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.200+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aa01372a0 con 0x7f5aa013f710 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.200+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5a940018a0 con 0x7f5aa013f710 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.200+0000 7f5a8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.200+0000 7f5a8effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.200+0000 7f5a8effd640 10 monclient(hunting): dump: 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:50.201 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.202 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.202 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.202 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:50.202 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:50.202 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:50.202 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:50.202 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3642968426 0 0) 0x7f5aa0135ac0 con 0x7f5aa01430c0 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 >> v1:172.21.15.18:6790/0 conn(0x7f5aa00a98f0 legacy=0x7f5aa0133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 >> v1:172.21.15.18:6789/0 conn(0x7f5aa013f710 legacy=0x7f5aa01379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 1 monclient: found mon.b 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa0137580 con 0x7f5aa01430c0 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 10 monclient: _finish_auth 0 2024-03-25T19:51:50.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5a8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 10 monclient: _renew_subs 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa01363b0 con 0x7f5aa01430c0 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 10 monclient: _renew_subs 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa01353a0 con 0x7f5aa01430c0 2024-03-25T19:51:50.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.217+0000 7f5aa4e9a640 1 librados: init done 2024-03-25T19:51:50.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.218+0000 7f5aa4e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:50.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.218+0000 7f5aa4e9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:50.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.218+0000 7f5aa4e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:50.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.218+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa0130d90 con 0x7f5aa01430c0 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.495+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a98003990 con 0x7f5aa01430c0 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.495+0000 7f5a8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.495+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f5a98003b80 con 0x7f5aa01430c0 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.495+0000 7f5a8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.495+0000 7f5a8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.495+0000 7f5a8effd640 10 monclient: dump: 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:50.496 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.495+0000 7f5a9cd99640 4 set_mon_vals no callback set 2024-03-25T19:51:50.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.496+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f5a98003da0 con 0x7f5aa01430c0 2024-03-25T19:51:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.497+0000 7f5a9e59c640 10 monclient: get_auth_request con 0x7f5a8006dfb0 auth_method 0 2024-03-25T19:51:50.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.497+0000 7f5a8effd640 10 client.4690.objecter ms_handle_connect 0x7f5a8006dfb0 2024-03-25T19:51:50.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.502+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7f5a98054840 con 0x7f5aa01430c0 2024-03-25T19:51:50.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.502+0000 7f5a8effd640 10 client.4690.objecter ms_dispatch 0x7f5aa0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:50.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.502+0000 7f5a8effd640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:50.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.502+0000 7f5a8effd640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:50.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.502+0000 7f5a8effd640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T19:51:50.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.502+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a98050350 con 0x7f5aa01430c0 2024-03-25T19:51:50.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.502+0000 7f5a8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:50.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.502+0000 7f5a8effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:50.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.753+0000 7f5aa4e9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.753+0000 7f5aa4e9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.753+0000 7f5aa4e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:51:50.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.753+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aa0074cc0 con 0x7f5aa01430c0 2024-03-25T19:51:50.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.790+0000 7f5a8effd640 1 -- 172.21.15.18:0/1599514233 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7f5a98052280 con 0x7f5aa01430c0 2024-03-25T19:51:50.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.790+0000 7f5a8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.790+0000 7f5a8effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:50.791 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:51:50.791 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:50.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 10 librados: watch_flush enter 2024-03-25T19:51:50.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 10 librados: watch_flush exit 2024-03-25T19:51:50.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T19:51:50.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 10 client.4690.objecter successfully canceled tick 2024-03-25T19:51:50.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 >> v1:172.21.15.18:6800/1520000573 conn(0x7f5a8006dfb0 legacy=0x7f5a80070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 10 monclient: shutdown 2024-03-25T19:51:50.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:50.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 >> v1:172.21.15.53:6789/0 conn(0x7f5aa01430c0 legacy=0x7f5aa01e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 shutdown_connections 2024-03-25T19:51:50.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 >> 172.21.15.18:0/1599514233 conn(0x7f5aa00b7ea0 msgr2=0x7f5aa0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:50.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 shutdown_connections 2024-03-25T19:51:50.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 1 -- 172.21.15.18:0/1599514233 wait complete. 2024-03-25T19:51:50.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:50.793+0000 7f5aa4e9a640 1 librados: shutdown 2024-03-25T19:51:50.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:51.313 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:18.844 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.321 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:18.852 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.329 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:18.825 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.330 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:18.861 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.331 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:18.828 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.333 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:18.864 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.346 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:18.876 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.353 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:18.849 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.356 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:18.887 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.613 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.726+0000 7f202d2e8640 10 monclient: get_monmap_and_config 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.726+0000 7f202d2e8640 10 monclient: build_initial_monmap 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.726+0000 7f202d2e8640 10 monclient: monmap: 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:51:57.727001+0000 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:51:57.727001+0000 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:57.726 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:57.727 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:57.727 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:51:57.727 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:51:57.727 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:57.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.726+0000 7f202d2e8640 1 Processor -- start 2024-03-25T19:51:57.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.726+0000 7f202d2e8640 1 -- start start 2024-03-25T19:51:57.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.726+0000 7f202d2e8640 10 monclient: init 2024-03-25T19:51:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 10 monclient(hunting): picked mon.noname-c con 0x7f20280a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 10 monclient(hunting): picked mon.noname-b con 0x7f202813f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 10 monclient(hunting): picked mon.noname-a con 0x7f20281430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2028142520 con 0x7f20281430c0 2024-03-25T19:51:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20280b28a0 con 0x7f20280a98f0 2024-03-25T19:51:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20280b2a30 con 0x7f202813f710 2024-03-25T19:51:57.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:57.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.727+0000 7f202d2e8640 10 monclient(hunting): authenticate will time out at 5544.854364s 2024-03-25T19:51:57.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.734+0000 7f2026ffd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f20280a98f0 0x7f20280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:54388/0 (socket says 172.21.15.18:54388) 2024-03-25T19:51:57.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.734+0000 7f2026ffd640 1 -- 172.21.15.18:0/3454563768 learned_addr learned my addr 172.21.15.18:0/3454563768 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:57.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.743+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637768145 0 0) 0x7f2028142520 con 0x7f20281430c0 2024-03-25T19:51:57.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.743+0000 7f2025ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.743+0000 7f2025ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.743+0000 7f2025ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.743+0000 7f2025ffb640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:51:57.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.743+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2010003520 con 0x7f20281430c0 2024-03-25T19:51:57.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2523521543 0 0) 0x7f2010003520 con 0x7f20281430c0 2024-03-25T19:51:57.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 >> v1:172.21.15.18:6790/0 conn(0x7f202813f710 legacy=0x7f20281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 >> v1:172.21.15.53:6789/0 conn(0x7f20280a98f0 legacy=0x7f20280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 1 monclient: found mon.noname-a 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20280b44c0 con 0x7f20281430c0 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2020001690 con 0x7f20281430c0 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.749+0000 7f2025ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.750+0000 7f2025ffb640 10 monclient: dump: 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:57.750 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.750+0000 7f202d2e8640 5 monclient: authenticate success, global_id 4716 2024-03-25T19:51:57.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.750+0000 7f202d2e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:57.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.758+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20200017f0 con 0x7f20281430c0 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.758+0000 7f2025ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.758+0000 7f2025ffb640 1 -- 172.21.15.18:0/3454563768 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2020001c00 con 0x7f20281430c0 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.758+0000 7f2025ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.758+0000 7f2025ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.758+0000 7f2025ffb640 10 monclient: dump: 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:57.759 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:57.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.759+0000 7f202d2e8640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:57.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.759+0000 7f202d2e8640 4 set_mon_vals no callback set 2024-03-25T19:51:57.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.759+0000 7f202d2e8640 10 monclient: shutdown 2024-03-25T19:51:57.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.759+0000 7f202d2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:57.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.759+0000 7f202d2e8640 1 -- 172.21.15.18:0/3454563768 >> v1:172.21.15.18:6789/0 conn(0x7f20281430c0 legacy=0x7f202814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.759+0000 7f202d2e8640 1 -- 172.21.15.18:0/3454563768 shutdown_connections 2024-03-25T19:51:57.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.759+0000 7f202d2e8640 1 -- 172.21.15.18:0/3454563768 >> 172.21.15.18:0/3454563768 conn(0x7f20280b7ea0 msgr2=0x7f20280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:57.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.760+0000 7f202d2e8640 1 -- 172.21.15.18:0/3454563768 shutdown_connections 2024-03-25T19:51:57.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.760+0000 7f202d2e8640 1 -- 172.21.15.18:0/3454563768 wait complete. 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.760+0000 7f202d2e8640 10 monclient: build_initial_monmap 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.760+0000 7f202d2e8640 10 monclient: monmap: 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.761+0000 7f202d2e8640 1 librados: starting msgr at 2024-03-25T19:51:57.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.761+0000 7f202d2e8640 1 librados: starting objecter 2024-03-25T19:51:57.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.761+0000 7f202d2e8640 1 Processor -- start 2024-03-25T19:51:57.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.761+0000 7f202d2e8640 1 -- start start 2024-03-25T19:51:57.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.761+0000 7f202d2e8640 1 librados: setting wanted keys 2024-03-25T19:51:57.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.761+0000 7f202d2e8640 1 librados: calling monclient init 2024-03-25T19:51:57.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.761+0000 7f202d2e8640 10 monclient: init 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 10 monclient(hunting): picked mon.noname-c con 0x7f20280a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 10 monclient(hunting): picked mon.noname-b con 0x7f202813f710 addr v1:172.21.15.53:6789/0 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 10 monclient(hunting): picked mon.noname-a con 0x7f20281430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20281df2b0 con 0x7f20281430c0 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f202806f7d0 con 0x7f202813f710 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20281dfee0 con 0x7f20280a98f0 2024-03-25T19:51:57.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:57.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.762+0000 7f202d2e8640 10 monclient(hunting): authenticate will time out at 5544.889361s 2024-03-25T19:51:57.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.766+0000 7f20267fc640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f202813f710 0x7f20281df7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:54426/0 (socket says 172.21.15.18:54426) 2024-03-25T19:51:57.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.766+0000 7f20267fc640 1 -- 172.21.15.18:0/2440697440 learned_addr learned my addr 172.21.15.18:0/2440697440 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:51:57.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.772+0000 7f200f7fe640 10 client.?.objecter ms_handle_connect 0x7f202813f710 2024-03-25T19:51:57.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.772+0000 7f200f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.773+0000 7f200f7fe640 10 client.?.objecter ms_handle_connect 0x7f20280a98f0 2024-03-25T19:51:57.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.773+0000 7f200f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.782+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155701476 0 0) 0x7f202806f7d0 con 0x7f202813f710 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.782+0000 7f200f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.782+0000 7f200f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.782+0000 7f200f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.782+0000 7f200f7fe640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.782+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2014004eb0 con 0x7f202813f710 2024-03-25T19:51:57.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.784+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310726491 0 0) 0x7f20281dfee0 con 0x7f20280a98f0 2024-03-25T19:51:57.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.784+0000 7f200f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.784+0000 7f200f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.784+0000 7f200f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.784+0000 7f200f7fe640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:51:57.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.784+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f202806f7d0 con 0x7f20280a98f0 2024-03-25T19:51:57.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.790+0000 7f200f7fe640 10 client.?.objecter ms_handle_connect 0x7f20281430c0 2024-03-25T19:51:57.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.790+0000 7f200f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.800+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1588280361 0 0) 0x7f202806f7d0 con 0x7f20280a98f0 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.800+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20281dfee0 con 0x7f20280a98f0 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.800+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2018001cb0 con 0x7f20280a98f0 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.800+0000 7f200f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.800+0000 7f200f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.800+0000 7f200f7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.800+0000 7f200f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:57.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.810+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 234131535 0 0) 0x7f2014004eb0 con 0x7f202813f710 2024-03-25T19:51:57.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.810+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f202806f7d0 con 0x7f202813f710 2024-03-25T19:51:57.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.810+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f201c001a30 con 0x7f202813f710 2024-03-25T19:51:57.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.810+0000 7f200f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.810+0000 7f200f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:57.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.810+0000 7f200f7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:57.810 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:57.811 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:57.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.811+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410456194 0 0) 0x7f20281df2b0 con 0x7f20281430c0 2024-03-25T19:51:57.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.812+0000 7f200f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.812+0000 7f200f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.812+0000 7f200f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.812+0000 7f200f7fe640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:51:57.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.812+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2014004eb0 con 0x7f20281430c0 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 690850355 0 0) 0x7f20281dfee0 con 0x7f20280a98f0 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 >> v1:172.21.15.53:6789/0 conn(0x7f202813f710 legacy=0x7f20281df7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 >> v1:172.21.15.18:6789/0 conn(0x7f20281430c0 legacy=0x7f20281e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 1 monclient: found mon.c 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20281e0070 con 0x7f20280a98f0 2024-03-25T19:51:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f200f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f202d2e8640 5 monclient: authenticate success, global_id 4694 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f202d2e8640 10 monclient: _renew_subs 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f202d2e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.822+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2028070dc0 con 0x7f20280a98f0 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.823+0000 7f202d2e8640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.823+0000 7f202d2e8640 10 monclient: _renew_subs 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.823+0000 7f202d2e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.823+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f202806ef70 con 0x7f20280a98f0 2024-03-25T19:51:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.823+0000 7f202d2e8640 1 librados: init done 2024-03-25T19:51:57.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.824+0000 7f202d2e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.824+0000 7f202d2e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.824+0000 7f202d2e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:57.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:57.824+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20280627b0 con 0x7f20280a98f0 2024-03-25T19:51:58.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.134+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20180020c0 con 0x7f20280a98f0 2024-03-25T19:51:58.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.134+0000 7f200f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:58.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.134+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7f2018004d40 con 0x7f20280a98f0 2024-03-25T19:51:58.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.134+0000 7f200f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:58.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.134+0000 7f200f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:58.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.134+0000 7f200f7fe640 10 monclient: dump: 2024-03-25T19:51:58.134 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.134+0000 7f20257fa640 4 set_mon_vals no callback set 2024-03-25T19:51:58.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.134+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7f2018004f60 con 0x7f20280a98f0 2024-03-25T19:51:58.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.135+0000 7f20267fc640 10 monclient: get_auth_request con 0x7f201406df40 auth_method 0 2024-03-25T19:51:58.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.135+0000 7f200f7fe640 10 client.4694.objecter ms_handle_connect 0x7f201406df40 2024-03-25T19:51:58.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.140+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7f2018054b80 con 0x7f20280a98f0 2024-03-25T19:51:58.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.140+0000 7f200f7fe640 10 client.4694.objecter ms_dispatch 0x7f2028000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:58.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.140+0000 7f200f7fe640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:58.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.140+0000 7f200f7fe640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:58.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.140+0000 7f200f7fe640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T19:51:58.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.140+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20180504c0 con 0x7f20280a98f0 2024-03-25T19:51:58.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.141+0000 7f200f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:58.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.141+0000 7f200f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:58.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.398+0000 7f202d2e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.398+0000 7f202d2e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.398+0000 7f202d2e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:51:58.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.398+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f202814e010 con 0x7f20280a98f0 2024-03-25T19:51:58.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.492+0000 7f200f7fe640 1 -- 172.21.15.18:0/2440697440 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7f20180525c0 con 0x7f20280a98f0 2024-03-25T19:51:58.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.492+0000 7f200f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.492+0000 7f200f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:58.492 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:51:58.493 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:58.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 10 librados: watch_flush enter 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 10 librados: watch_flush exit 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 10 client.4694.objecter successfully canceled tick 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 >> v1:172.21.15.18:6800/1520000573 conn(0x7f201406df40 legacy=0x7f2014070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 10 monclient: shutdown 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 >> v1:172.21.15.18:6790/0 conn(0x7f20280a98f0 legacy=0x7f202806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 shutdown_connections 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.495+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 >> 172.21.15.18:0/2440697440 conn(0x7f20280b7ea0 msgr2=0x7f2028074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.496+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 shutdown_connections 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.496+0000 7f202d2e8640 1 -- 172.21.15.18:0/2440697440 wait complete. 2024-03-25T19:51:58.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:51:58.496+0000 7f202d2e8640 1 librados: shutdown 2024-03-25T19:51:58.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:04.147 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:31.678 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.196 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:31.693 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.198 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:31.694 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.206 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:31.736 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.229 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:31.726 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.236 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:31.767 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.266 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:31.797 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.272 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:31.802 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:04.304 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:31.835 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:05.313 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:05.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.426+0000 7ffa73e74640 10 monclient: get_monmap_and_config 2024-03-25T19:52:05.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.426+0000 7ffa73e74640 10 monclient: build_initial_monmap 2024-03-25T19:52:05.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.426+0000 7ffa73e74640 10 monclient: monmap: 2024-03-25T19:52:05.426 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:52:05.427558+0000 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:52:05.427558+0000 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:52:05.427 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:05.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.427+0000 7ffa73e74640 1 Processor -- start 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.427+0000 7ffa73e74640 1 -- start start 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.427+0000 7ffa73e74640 10 monclient: init 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 10 monclient(hunting): picked mon.noname-c con 0x7ffa6c0b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 10 monclient(hunting): picked mon.noname-b con 0x7ffa6c0b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 10 monclient(hunting): picked mon.noname-a con 0x7ffa6c14be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c0af940 con 0x7ffa6c14be90 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c0b0a70 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c0b0d50 con 0x7ffa6c0b1730 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:05.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.428+0000 7ffa73e74640 10 monclient(hunting): authenticate will time out at 5552.555471s 2024-03-25T19:52:05.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.435+0000 7ffa72411640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7ffa6c14be90 0x7ffa6c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:33882/0 (socket says 172.21.15.18:33882) 2024-03-25T19:52:05.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.435+0000 7ffa72411640 1 -- 172.21.15.18:0/387088631 learned_addr learned my addr 172.21.15.18:0/387088631 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:52:05.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.447+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865023283 0 0) 0x7ffa6c0af940 con 0x7ffa6c14be90 2024-03-25T19:52:05.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.447+0000 7ffa70c0e640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.447+0000 7ffa70c0e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.447+0000 7ffa70c0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.447+0000 7ffa70c0e640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:52:05.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.447+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa58003520 con 0x7ffa6c14be90 2024-03-25T19:52:05.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1061483009 0 0) 0x7ffa58003520 con 0x7ffa6c14be90 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 >> v1:172.21.15.18:6790/0 conn(0x7ffa6c0b1730 legacy=0x7ffa6c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 >> v1:172.21.15.53:6789/0 conn(0x7ffa6c0b4d60 legacy=0x7ffa6c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 10 monclient: _finish_hunting 0 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 1 monclient: found mon.noname-a 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c0b1030 con 0x7ffa6c14be90 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 10 monclient: _finish_auth 0 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 10 monclient: _check_auth_tickets 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffa64001a10 con 0x7ffa6c14be90 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa73e74640 5 monclient: authenticate success, global_id 4722 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa73e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:05.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa70c0e640 10 monclient: dump: 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:05.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.455+0000 7ffa73e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:05.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.475+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa64001be0 con 0x7ffa6c14be90 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.475+0000 7ffa70c0e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.475+0000 7ffa70c0e640 1 -- 172.21.15.18:0/387088631 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffa64002030 con 0x7ffa6c14be90 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.475+0000 7ffa73e74640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.475+0000 7ffa73e74640 4 set_mon_vals no callback set 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.475+0000 7ffa70c0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.475+0000 7ffa70c0e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.475+0000 7ffa70c0e640 10 monclient: dump: 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:05.475 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:05.476 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:05.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.476+0000 7ffa73e74640 10 monclient: shutdown 2024-03-25T19:52:05.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.476+0000 7ffa73e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:05.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.476+0000 7ffa73e74640 1 -- 172.21.15.18:0/387088631 >> v1:172.21.15.18:6789/0 conn(0x7ffa6c14be90 legacy=0x7ffa6c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.476+0000 7ffa73e74640 1 -- 172.21.15.18:0/387088631 shutdown_connections 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.476+0000 7ffa73e74640 1 -- 172.21.15.18:0/387088631 >> 172.21.15.18:0/387088631 conn(0x7ffa6c0617a0 msgr2=0x7ffa6c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.476+0000 7ffa73e74640 1 -- 172.21.15.18:0/387088631 shutdown_connections 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.476+0000 7ffa73e74640 1 -- 172.21.15.18:0/387088631 wait complete. 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 10 monclient: build_initial_monmap 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 10 monclient: monmap: 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:52:05.477 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:05.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 1 librados: starting msgr at 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 1 librados: starting objecter 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 1 Processor -- start 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 1 -- start start 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 1 librados: setting wanted keys 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 1 librados: calling monclient init 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.477+0000 7ffa73e74640 10 monclient: init 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 10 monclient(hunting): picked mon.noname-c con 0x7ffa6c0b1730 addr v1:172.21.15.18:6790/0 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 10 monclient(hunting): picked mon.noname-b con 0x7ffa6c0b4d60 addr v1:172.21.15.53:6789/0 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 10 monclient(hunting): picked mon.noname-a con 0x7ffa6c14be90 addr v1:172.21.15.18:6789/0 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c13bad0 con 0x7ffa6c14be90 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c1d7810 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c1d8440 con 0x7ffa6c0b1730 2024-03-25T19:52:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:05.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.478+0000 7ffa73e74640 10 monclient(hunting): authenticate will time out at 5552.605307s 2024-03-25T19:52:05.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.487+0000 7ffa7140f640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ffa6c0b4d60 0x7ffa6c132db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:58646/0 (socket says 172.21.15.18:58646) 2024-03-25T19:52:05.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.487+0000 7ffa7140f640 1 -- 172.21.15.18:0/2552791361 learned_addr learned my addr 172.21.15.18:0/2552791361 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:52:05.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.488+0000 7ffa627fc640 10 client.?.objecter ms_handle_connect 0x7ffa6c0b4d60 2024-03-25T19:52:05.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.488+0000 7ffa627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.497+0000 7ffa627fc640 10 client.?.objecter ms_handle_connect 0x7ffa6c0b1730 2024-03-25T19:52:05.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.497+0000 7ffa627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.500+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821836896 0 0) 0x7ffa6c1d7810 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.500+0000 7ffa627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.500+0000 7ffa627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.500+0000 7ffa627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.500+0000 7ffa627fc640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:52:05.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.500+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa50004eb0 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.504+0000 7ffa627fc640 10 client.?.objecter ms_handle_connect 0x7ffa6c14be90 2024-03-25T19:52:05.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.504+0000 7ffa627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.511+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413646443 0 0) 0x7ffa6c13bad0 con 0x7ffa6c14be90 2024-03-25T19:52:05.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.511+0000 7ffa627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.511+0000 7ffa627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.511+0000 7ffa627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.511+0000 7ffa627fc640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:52:05.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.511+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa6c1d7810 con 0x7ffa6c14be90 2024-03-25T19:52:05.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.520+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427137497 0 0) 0x7ffa6c1d8440 con 0x7ffa6c0b1730 2024-03-25T19:52:05.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.520+0000 7ffa627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.520+0000 7ffa627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.520+0000 7ffa627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.520+0000 7ffa627fc640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:52:05.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.520+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa6c13bad0 con 0x7ffa6c0b1730 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.533+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2087348108 0 0) 0x7ffa50004eb0 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.533+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa6c1d8440 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.533+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffa5c001960 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.533+0000 7ffa627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.533+0000 7ffa627fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.533+0000 7ffa627fc640 10 monclient(hunting): dump: 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.533 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.534 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:05.534 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:05.534 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:05.534 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:05.534 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:05.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.533+0000 7ffa627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.541+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 764382583 0 0) 0x7ffa6c1d7810 con 0x7ffa6c14be90 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.541+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa50004eb0 con 0x7ffa6c14be90 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.541+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffa64001ef0 con 0x7ffa6c14be90 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.541+0000 7ffa627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.541+0000 7ffa627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.541+0000 7ffa627fc640 10 monclient(hunting): dump: 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:05.541 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:05.542 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:05.542 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:05.542 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:05.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.542+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 12731611 0 0) 0x7ffa6c13bad0 con 0x7ffa6c0b1730 2024-03-25T19:52:05.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.543+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa6c1d7810 con 0x7ffa6c0b1730 2024-03-25T19:52:05.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.543+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffa68001860 con 0x7ffa6c0b1730 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.543+0000 7ffa627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.543+0000 7ffa627fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.543+0000 7ffa627fc640 10 monclient(hunting): dump: 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:05.543 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:05.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 776564519 0 0) 0x7ffa6c1d8440 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 >> v1:172.21.15.18:6790/0 conn(0x7ffa6c0b1730 legacy=0x7ffa6c1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 >> v1:172.21.15.18:6789/0 conn(0x7ffa6c14be90 legacy=0x7ffa6c1e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 1 monclient: found mon.b 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c1d85d0 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 10 monclient: _finish_auth 0 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa73e74640 5 monclient: authenticate success, global_id 4702 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa73e74640 10 monclient: _renew_subs 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa73e74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:52:05.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.552+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.53:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c139400 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.553+0000 7ffa73e74640 10 client.4702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:05.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.553+0000 7ffa73e74640 10 monclient: _renew_subs 2024-03-25T19:52:05.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.553+0000 7ffa73e74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:52:05.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.553+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.53:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c13d2d0 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.553+0000 7ffa73e74640 1 librados: init done 2024-03-25T19:52:05.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.554+0000 7ffa73e74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.554+0000 7ffa73e74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.554+0000 7ffa73e74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:52:05.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.554+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6c0627b0 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.861+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa5c0035f0 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.861+0000 7ffa627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.861+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7ffa5c003940 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.861+0000 7ffa627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.861+0000 7ffa627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.861+0000 7ffa63fff640 4 set_mon_vals no callback set 2024-03-25T19:52:05.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.861+0000 7ffa627fc640 10 monclient: dump: 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:05.862 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:05.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.862+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7ffa5c003b60 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.863+0000 7ffa71c10640 10 monclient: get_auth_request con 0x7ffa5006e0b0 auth_method 0 2024-03-25T19:52:05.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.863+0000 7ffa627fc640 10 client.4702.objecter ms_handle_connect 0x7ffa5006e0b0 2024-03-25T19:52:05.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.868+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7ffa5c0547d0 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.868+0000 7ffa627fc640 10 client.4702.objecter ms_dispatch 0x7ffa6c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:05.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.868+0000 7ffa627fc640 3 client.4702.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:05.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.868+0000 7ffa627fc640 3 client.4702.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:05.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.868+0000 7ffa627fc640 20 client.4702.objecter dump_active .. 0 homeless 2024-03-25T19:52:05.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.868+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa5c0502e0 con 0x7ffa6c0b4d60 2024-03-25T19:52:05.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.868+0000 7ffa627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:05.868+0000 7ffa627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:06.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.098+0000 7ffa73e74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:06.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.098+0000 7ffa73e74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:06.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.098+0000 7ffa73e74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.53:6789/0 2024-03-25T19:52:06.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.098+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 --> v1:172.21.15.53:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa6c1d79a0 con 0x7ffa6c0b4d60 2024-03-25T19:52:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.245+0000 7ffa627fc640 1 -- 172.21.15.18:0/2552791361 <== mon.1 v1:172.21.15.53:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7ffa5c052210 con 0x7ffa6c0b4d60 2024-03-25T19:52:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.245+0000 7ffa627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.245+0000 7ffa627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:06.245 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:52:06.245 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.247+0000 7ffa73e74640 10 librados: watch_flush enter 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.247+0000 7ffa73e74640 10 librados: watch_flush exit 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.247+0000 7ffa73e74640 20 client.4702.objecter shutdown clearing up homeless session... 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.247+0000 7ffa73e74640 10 client.4702.objecter successfully canceled tick 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 >> v1:172.21.15.18:6800/1520000573 conn(0x7ffa5006e0b0 legacy=0x7ffa50070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 10 monclient: shutdown 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 >> v1:172.21.15.53:6789/0 conn(0x7ffa6c0b4d60 legacy=0x7ffa6c132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:06.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 shutdown_connections 2024-03-25T19:52:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 >> 172.21.15.18:0/2552791361 conn(0x7ffa6c0617a0 msgr2=0x7ffa6c0b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 shutdown_connections 2024-03-25T19:52:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 1 -- 172.21.15.18:0/2552791361 wait complete. 2024-03-25T19:52:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:06.248+0000 7ffa73e74640 1 librados: shutdown 2024-03-25T19:52:06.259 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:13.065 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.184+0000 7fed191c6640 10 monclient: get_monmap_and_config 2024-03-25T19:52:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.184+0000 7fed191c6640 10 monclient: build_initial_monmap 2024-03-25T19:52:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.184+0000 7fed191c6640 10 monclient: monmap: 2024-03-25T19:52:13.184 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:52:13.184 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:13.184 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:52:13.185602+0000 2024-03-25T19:52:13.184 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:52:13.185602+0000 2024-03-25T19:52:13.184 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:13.185 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:13.185 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:52:13.185 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:52:13.185 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:52:13.185 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:13.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.185+0000 7fed191c6640 1 Processor -- start 2024-03-25T19:52:13.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.185+0000 7fed191c6640 1 -- start start 2024-03-25T19:52:13.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.185+0000 7fed191c6640 10 monclient: init 2024-03-25T19:52:13.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 10 monclient(hunting): picked mon.noname-a con 0x7fed140a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 10 monclient(hunting): picked mon.noname-b con 0x7fed1413f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 10 monclient(hunting): picked mon.noname-c con 0x7fed141430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed14142520 con 0x7fed140a98f0 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed140b28a0 con 0x7fed141430c0 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed140b2a30 con 0x7fed1413f710 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.186+0000 7fed191c6640 10 monclient(hunting): authenticate will time out at 5560.313377s 2024-03-25T19:52:13.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.190+0000 7fed12d9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fed140a98f0 0x7fed140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:48492/0 (socket says 172.21.15.18:48492) 2024-03-25T19:52:13.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.190+0000 7fed12d9d640 1 -- 172.21.15.18:0/2327395031 learned_addr learned my addr 172.21.15.18:0/2327395031 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:52:13.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.204+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931255699 0 0) 0x7fed14142520 con 0x7fed140a98f0 2024-03-25T19:52:13.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.204+0000 7fed11d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.204+0000 7fed11d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.204+0000 7fed11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.204+0000 7fed11d9b640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:52:13.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.204+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecfc003520 con 0x7fed140a98f0 2024-03-25T19:52:13.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.207+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158580235 0 0) 0x7fed140b2a30 con 0x7fed1413f710 2024-03-25T19:52:13.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.207+0000 7fed11d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.207+0000 7fed11d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.207+0000 7fed11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.207+0000 7fed11d9b640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:52:13.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.207+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed14142520 con 0x7fed1413f710 2024-03-25T19:52:13.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.211+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082290085 0 0) 0x7fed140b28a0 con 0x7fed141430c0 2024-03-25T19:52:13.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.211+0000 7fed11d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.211+0000 7fed11d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.211+0000 7fed11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.211+0000 7fed11d9b640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:52:13.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.211+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed140b2a30 con 0x7fed141430c0 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 706545211 0 0) 0x7fecfc003520 con 0x7fed140a98f0 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 >> v1:172.21.15.18:6790/0 conn(0x7fed1413f710 legacy=0x7fed141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 >> v1:172.21.15.53:6789/0 conn(0x7fed141430c0 legacy=0x7fed1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 1 monclient: found mon.noname-a 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed140b44c0 con 0x7fed140a98f0 2024-03-25T19:52:13.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 10 monclient: _finish_auth 0 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fed04001690 con 0x7fed140a98f0 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed11d9b640 10 monclient: dump: 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:13.231 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:13.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed191c6640 5 monclient: authenticate success, global_id 4728 2024-03-25T19:52:13.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.230+0000 7fed191c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:13.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.244+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed040017f0 con 0x7fed140a98f0 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.244+0000 7fed11d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.244+0000 7fed11d9b640 1 -- 172.21.15.18:0/2327395031 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fed04001c00 con 0x7fed140a98f0 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.244+0000 7fed191c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.244+0000 7fed191c6640 4 set_mon_vals no callback set 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.244+0000 7fed11d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.244+0000 7fed11d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.244+0000 7fed11d9b640 10 monclient: dump: 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:13.244 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:13.245 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:13.245 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:13.245 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.245+0000 7fed191c6640 10 monclient: shutdown 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.245+0000 7fed191c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.245+0000 7fed191c6640 1 -- 172.21.15.18:0/2327395031 >> v1:172.21.15.18:6789/0 conn(0x7fed140a98f0 legacy=0x7fed140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.245+0000 7fed191c6640 1 -- 172.21.15.18:0/2327395031 shutdown_connections 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.245+0000 7fed191c6640 1 -- 172.21.15.18:0/2327395031 >> 172.21.15.18:0/2327395031 conn(0x7fed140b7ea0 msgr2=0x7fed140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.245+0000 7fed191c6640 1 -- 172.21.15.18:0/2327395031 shutdown_connections 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.245+0000 7fed191c6640 1 -- 172.21.15.18:0/2327395031 wait complete. 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.246+0000 7fed191c6640 10 monclient: build_initial_monmap 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.246+0000 7fed191c6640 10 monclient: monmap: 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:52:13.246 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.246+0000 7fed191c6640 1 librados: starting msgr at 2024-03-25T19:52:13.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.246+0000 7fed191c6640 1 librados: starting objecter 2024-03-25T19:52:13.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.247+0000 7fed191c6640 1 Processor -- start 2024-03-25T19:52:13.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.247+0000 7fed191c6640 1 -- start start 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.247+0000 7fed191c6640 1 librados: setting wanted keys 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.247+0000 7fed191c6640 1 librados: calling monclient init 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.247+0000 7fed191c6640 10 monclient: init 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.247+0000 7fed191c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.247+0000 7fed191c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.247+0000 7fed191c6640 10 monclient(hunting): picked mon.noname-b con 0x7fed140a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.248+0000 7fed191c6640 10 monclient(hunting): picked mon.noname-a con 0x7fed1413f710 addr v1:172.21.15.18:6789/0 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.248+0000 7fed191c6640 10 monclient(hunting): picked mon.noname-c con 0x7fed141430c0 addr v1:172.21.15.18:6790/0 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.248+0000 7fed191c6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed140b7ce0 con 0x7fed1413f710 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.248+0000 7fed191c6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed141319b0 con 0x7fed140a98f0 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.248+0000 7fed191c6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed140b7800 con 0x7fed141430c0 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.248+0000 7fed191c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.248+0000 7fed191c6640 10 monclient(hunting): authenticate will time out at 5560.374901s 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.252+0000 7fed12d9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fed140a98f0 0x7fed141dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:56564/0 (socket says 172.21.15.18:56564) 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.252+0000 7fed12d9d640 1 -- 172.21.15.18:0/1637807834 learned_addr learned my addr 172.21.15.18:0/1637807834 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:52:13.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.258+0000 7fecfb7fe640 10 client.?.objecter ms_handle_connect 0x7fed140a98f0 2024-03-25T19:52:13.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.258+0000 7fecfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:13.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.261+0000 7fecfb7fe640 10 client.?.objecter ms_handle_connect 0x7fed141430c0 2024-03-25T19:52:13.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.261+0000 7fecfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:13.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.263+0000 7fecfb7fe640 10 client.?.objecter ms_handle_connect 0x7fed1413f710 2024-03-25T19:52:13.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.263+0000 7fecfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:13.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.268+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891795509 0 0) 0x7fed141319b0 con 0x7fed140a98f0 2024-03-25T19:52:13.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.268+0000 7fecfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.268+0000 7fecfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.268+0000 7fecfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.268+0000 7fecfb7fe640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:52:13.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.268+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed00004eb0 con 0x7fed140a98f0 2024-03-25T19:52:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.270+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85811203 0 0) 0x7fed140b7800 con 0x7fed141430c0 2024-03-25T19:52:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.270+0000 7fecfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.270+0000 7fecfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.271+0000 7fecfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.271+0000 7fecfb7fe640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:52:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.271+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed141319b0 con 0x7fed141430c0 2024-03-25T19:52:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.279+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627148990 0 0) 0x7fed140b7ce0 con 0x7fed1413f710 2024-03-25T19:52:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.279+0000 7fecfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.279+0000 7fecfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.279+0000 7fecfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.279+0000 7fecfb7fe640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:52:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.279+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed140b7800 con 0x7fed1413f710 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.290+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2133076349 0 0) 0x7fed141319b0 con 0x7fed141430c0 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.290+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed140b7ce0 con 0x7fed141430c0 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.290+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fed0c001b50 con 0x7fed141430c0 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.290+0000 7fecfb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.290+0000 7fecfb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.290+0000 7fecfb7fe640 10 monclient(hunting): dump: 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:13.290 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:13.291 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:13.291 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:13.291 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:13.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.290+0000 7fecfb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:13.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.293+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2014080493 0 0) 0x7fed140b7800 con 0x7fed1413f710 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.293+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed141319b0 con 0x7fed1413f710 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.293+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fed080018f0 con 0x7fed1413f710 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.293+0000 7fecfb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.293+0000 7fecfb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.293+0000 7fecfb7fe640 10 monclient(hunting): dump: 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:13.293 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:13.294 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:13.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.295+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3893745592 0 0) 0x7fed00004eb0 con 0x7fed140a98f0 2024-03-25T19:52:13.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.295+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed140b7800 con 0x7fed140a98f0 2024-03-25T19:52:13.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.295+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fed04001bc0 con 0x7fed140a98f0 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.295+0000 7fecfb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.295+0000 7fecfb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.295+0000 7fecfb7fe640 10 monclient(hunting): dump: 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:13.295 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:13.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1234829911 0 0) 0x7fed141319b0 con 0x7fed1413f710 2024-03-25T19:52:13.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 >> v1:172.21.15.18:6790/0 conn(0x7fed141430c0 legacy=0x7fed141312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 >> v1:172.21.15.53:6789/0 conn(0x7fed140a98f0 legacy=0x7fed141dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:52:13.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 1 monclient: found mon.a 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed14136f50 con 0x7fed1413f710 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fecfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 5 monclient: authenticate success, global_id 4731 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 10 monclient: _renew_subs 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed141350f0 con 0x7fed1413f710 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 10 monclient: _renew_subs 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed14135c60 con 0x7fed1413f710 2024-03-25T19:52:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.302+0000 7fed191c6640 1 librados: init done 2024-03-25T19:52:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.303+0000 7fed191c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.303+0000 7fed191c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.303+0000 7fed191c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:52:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.303+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed140627b0 con 0x7fed1413f710 2024-03-25T19:52:13.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.520+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed080035f0 con 0x7fed1413f710 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.520+0000 7fecfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.520+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fed08001d80 con 0x7fed1413f710 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.520+0000 7fecfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.520+0000 7fecfb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.520+0000 7fecfb7fe640 10 monclient: dump: 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:13.520 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:13.521 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:13.521 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:13.521 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:13.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.520+0000 7fed1159a640 4 set_mon_vals no callback set 2024-03-25T19:52:13.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.628+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fed08019770 con 0x7fed1413f710 2024-03-25T19:52:13.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.629+0000 7fed12d9d640 10 monclient: get_auth_request con 0x7fed0006e0c0 auth_method 0 2024-03-25T19:52:13.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.629+0000 7fecfb7fe640 10 client.4731.objecter ms_handle_connect 0x7fed0006e0c0 2024-03-25T19:52:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.634+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7fed08053530 con 0x7fed1413f710 2024-03-25T19:52:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.634+0000 7fecfb7fe640 10 client.4731.objecter ms_dispatch 0x7fed14000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.634+0000 7fecfb7fe640 3 client.4731.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.634+0000 7fecfb7fe640 3 client.4731.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:13.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.634+0000 7fecfb7fe640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T19:52:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.634+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed0804ee70 con 0x7fed1413f710 2024-03-25T19:52:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.634+0000 7fecfb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:13.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.634+0000 7fecfb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:13.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.891+0000 7fed191c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.891+0000 7fed191c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.891+0000 7fed191c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T19:52:13.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.891+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed140b7a00 con 0x7fed1413f710 2024-03-25T19:52:13.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.918+0000 7fecfb7fe640 1 -- 172.21.15.18:0/1637807834 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7fed140b7a00 con 0x7fed1413f710 2024-03-25T19:52:13.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.918+0000 7fecfb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.918+0000 7fecfb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:13.918 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:52:13.918 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 10 librados: watch_flush enter 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 10 librados: watch_flush exit 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 10 client.4731.objecter successfully canceled tick 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 >> v1:172.21.15.18:6800/1520000573 conn(0x7fed0006e0c0 legacy=0x7fed00070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 10 monclient: shutdown 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 >> v1:172.21.15.18:6789/0 conn(0x7fed1413f710 legacy=0x7fed14130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 shutdown_connections 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 >> 172.21.15.18:0/1637807834 conn(0x7fed140b7ea0 msgr2=0x7fed14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 shutdown_connections 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.920+0000 7fed191c6640 1 -- 172.21.15.18:0/1637807834 wait complete. 2024-03-25T19:52:13.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:13.921+0000 7fed191c6640 1 librados: shutdown 2024-03-25T19:52:13.932 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:20.737 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:20.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.852+0000 7fbe94d4d640 10 monclient: get_monmap_and_config 2024-03-25T19:52:20.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.852+0000 7fbe94d4d640 10 monclient: build_initial_monmap 2024-03-25T19:52:20.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.852+0000 7fbe94d4d640 10 monclient: monmap: 2024-03-25T19:52:20.852 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:52:20.852 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:20.853 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:52:20.853562+0000 2024-03-25T19:52:20.853 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:52:20.853562+0000 2024-03-25T19:52:20.853 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:20.853 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:20.853 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:52:20.853 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-c 2024-03-25T19:52:20.853 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T19:52:20.853 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:20.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.853+0000 7fbe94d4d640 1 Processor -- start 2024-03-25T19:52:20.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.853+0000 7fbe94d4d640 1 -- start start 2024-03-25T19:52:20.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.853+0000 7fbe94d4d640 10 monclient: init 2024-03-25T19:52:20.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe900a98f0 addr v1:172.21.15.53:6789/0 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe9013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe901430c0 addr v1:172.21.15.18:6789/0 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90142520 con 0x7fbe901430c0 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe900b28a0 con 0x7fbe900a98f0 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe900b2a30 con 0x7fbe9013f710 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:20.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.854+0000 7fbe94d4d640 10 monclient(hunting): authenticate will time out at 5567.981640s 2024-03-25T19:52:20.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.859+0000 7fbe8e59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbe900a98f0 0x7fbe900a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:59950/0 (socket says 172.21.15.18:59950) 2024-03-25T19:52:20.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.860+0000 7fbe8e59c640 1 -- 172.21.15.18:0/3590653749 learned_addr learned my addr 172.21.15.18:0/3590653749 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:52:20.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.875+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936026903 0 0) 0x7fbe900b28a0 con 0x7fbe900a98f0 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.875+0000 7fbe8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.875+0000 7fbe8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.875+0000 7fbe8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.875+0000 7fbe8dd9b640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.875+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe7c003520 con 0x7fbe900a98f0 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.885+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724286191 0 0) 0x7fbe90142520 con 0x7fbe901430c0 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.885+0000 7fbe8dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.885+0000 7fbe8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.885+0000 7fbe8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.885+0000 7fbe8dd9b640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:52:20.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.885+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe900b28a0 con 0x7fbe901430c0 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 480755802 0 0) 0x7fbe7c003520 con 0x7fbe900a98f0 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 >> v1:172.21.15.18:6790/0 conn(0x7fbe9013f710 legacy=0x7fbe901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 >> v1:172.21.15.18:6789/0 conn(0x7fbe901430c0 legacy=0x7fbe9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 1 monclient: found mon.noname-c 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.53:6789/0 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe900b44c0 con 0x7fbe900a98f0 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbe80001690 con 0x7fbe900a98f0 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe8dd9b640 10 monclient: dump: 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe94d4d640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.887+0000 7fbe94d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:20.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 <== mon.1 v1:172.21.15.53:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe800017f0 con 0x7fbe900a98f0 2024-03-25T19:52:20.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe8dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:20.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe8dd9b640 1 -- 172.21.15.18:0/3590653749 <== mon.1 v1:172.21.15.53:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbe80001c00 con 0x7fbe900a98f0 2024-03-25T19:52:20.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe94d4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe94d4d640 4 set_mon_vals no callback set 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe8dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe8dd9b640 10 monclient: dump: 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:20.900 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe94d4d640 10 monclient: shutdown 2024-03-25T19:52:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe94d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.900+0000 7fbe94d4d640 1 -- 172.21.15.18:0/3590653749 >> v1:172.21.15.53:6789/0 conn(0x7fbe900a98f0 legacy=0x7fbe900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.901+0000 7fbe94d4d640 1 -- 172.21.15.18:0/3590653749 shutdown_connections 2024-03-25T19:52:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.901+0000 7fbe94d4d640 1 -- 172.21.15.18:0/3590653749 >> 172.21.15.18:0/3590653749 conn(0x7fbe900b7ea0 msgr2=0x7fbe900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.901+0000 7fbe94d4d640 1 -- 172.21.15.18:0/3590653749 shutdown_connections 2024-03-25T19:52:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.901+0000 7fbe94d4d640 1 -- 172.21.15.18:0/3590653749 wait complete. 2024-03-25T19:52:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 10 monclient: build_initial_monmap 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 10 monclient: monmap: 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.noname-b 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 1 librados: starting msgr at 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 1 librados: starting objecter 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 1 Processor -- start 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 1 -- start start 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 1 librados: setting wanted keys 2024-03-25T19:52:20.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 1 librados: calling monclient init 2024-03-25T19:52:20.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.902+0000 7fbe94d4d640 10 monclient: init 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 10 monclient(hunting): picked mon.noname-a con 0x7fbe900a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 10 monclient(hunting): picked mon.noname-c con 0x7fbe9013f710 addr v1:172.21.15.18:6790/0 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 10 monclient(hunting): picked mon.noname-b con 0x7fbe901430c0 addr v1:172.21.15.53:6789/0 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90139870 con 0x7fbe900a98f0 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe901d51f0 con 0x7fbe901430c0 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe90138830 con 0x7fbe9013f710 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:20.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.903+0000 7fbe94d4d640 10 monclient(hunting): authenticate will time out at 5568.030290s 2024-03-25T19:52:20.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.915+0000 7fbe8ed9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbe901430c0 0x7fbe901e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.18:59974/0 (socket says 172.21.15.18:59974) 2024-03-25T19:52:20.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.915+0000 7fbe8ed9d640 1 -- 172.21.15.18:0/540088892 learned_addr learned my addr 172.21.15.18:0/540088892 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T19:52:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.915+0000 7fbe86ffd640 10 client.?.objecter ms_handle_connect 0x7fbe901430c0 2024-03-25T19:52:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.915+0000 7fbe86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:20.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.917+0000 7fbe86ffd640 10 client.?.objecter ms_handle_connect 0x7fbe9013f710 2024-03-25T19:52:20.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.917+0000 7fbe86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:20.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.929+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125676023 0 0) 0x7fbe90138830 con 0x7fbe9013f710 2024-03-25T19:52:20.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.929+0000 7fbe86ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.929+0000 7fbe86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.929+0000 7fbe86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.929+0000 7fbe86ffd640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:52:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.929+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe70004eb0 con 0x7fbe9013f710 2024-03-25T19:52:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.935+0000 7fbe86ffd640 10 client.?.objecter ms_handle_connect 0x7fbe900a98f0 2024-03-25T19:52:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.935+0000 7fbe86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.940+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681003894 0 0) 0x7fbe901d51f0 con 0x7fbe901430c0 2024-03-25T19:52:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.940+0000 7fbe86ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.940+0000 7fbe86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.940+0000 7fbe86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.940+0000 7fbe86ffd640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:52:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.940+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe90138830 con 0x7fbe901430c0 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.948+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 83221426 0 0) 0x7fbe70004eb0 con 0x7fbe9013f710 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.948+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe901d51f0 con 0x7fbe9013f710 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.948+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbe780018c0 con 0x7fbe9013f710 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.948+0000 7fbe86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.948+0000 7fbe86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.948+0000 7fbe86ffd640 10 monclient(hunting): dump: 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:20.949 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:20.949 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:20.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.948+0000 7fbe86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:20.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.958+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 318366133 0 0) 0x7fbe90138830 con 0x7fbe901430c0 2024-03-25T19:52:20.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.958+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.53:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe70004eb0 con 0x7fbe901430c0 2024-03-25T19:52:20.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.958+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbe88001c10 con 0x7fbe901430c0 2024-03-25T19:52:20.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.958+0000 7fbe86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.958+0000 7fbe86ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:20.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.958+0000 7fbe86ffd640 10 monclient(hunting): dump: 2024-03-25T19:52:20.958 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:20.959 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2597840783 0 0) 0x7fbe901d51f0 con 0x7fbe9013f710 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 >> v1:172.21.15.53:6789/0 conn(0x7fbe901430c0 legacy=0x7fbe901e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 >> v1:172.21.15.18:6789/0 conn(0x7fbe900a98f0 legacy=0x7fbe90138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 1 monclient: found mon.c 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe901d77a0 con 0x7fbe9013f710 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 10 monclient: _finish_auth 0 2024-03-25T19:52:20.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 5 monclient: authenticate success, global_id 4709 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 10 monclient: _renew_subs 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe901d83d0 con 0x7fbe9013f710 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 10 monclient: _renew_subs 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe90138c80 con 0x7fbe9013f710 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.960+0000 7fbe94d4d640 1 librados: init done 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.961+0000 7fbe94d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.961+0000 7fbe94d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.961+0000 7fbe94d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:20.961+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe900b3e80 con 0x7fbe9013f710 2024-03-25T19:52:21.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.321+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe780035f0 con 0x7fbe9013f710 2024-03-25T19:52:21.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.321+0000 7fbe86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.321+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 103208422 0 0) 0x7fbe78003900 con 0x7fbe9013f710 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.321+0000 7fbe86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.321+0000 7fbe86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.321+0000 7fbe86ffd640 10 monclient: dump: 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:fsid e9758276-eabb-4964-9819-31fbe65d0466 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T19:40:03.502906+0000 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.53:6789/0 mon.b 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T19:52:21.322 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T19:52:21.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.321+0000 7fbe8d59a640 4 set_mon_vals no callback set 2024-03-25T19:52:21.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.322+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3584277661 0 0) 0x7fbe780198d0 con 0x7fbe9013f710 2024-03-25T19:52:21.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.323+0000 7fbe8e59c640 10 monclient: get_auth_request con 0x7fbe7006de90 auth_method 0 2024-03-25T19:52:21.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.323+0000 7fbe86ffd640 10 client.4709.objecter ms_handle_connect 0x7fbe7006de90 2024-03-25T19:52:21.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.330+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1124934350 0 0) 0x7fbe78053570 con 0x7fbe9013f710 2024-03-25T19:52:21.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.330+0000 7fbe86ffd640 10 client.4709.objecter ms_dispatch 0x7fbe90000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:21.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.330+0000 7fbe86ffd640 3 client.4709.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:21.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.330+0000 7fbe86ffd640 3 client.4709.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:21.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.330+0000 7fbe86ffd640 20 client.4709.objecter dump_active .. 0 homeless 2024-03-25T19:52:21.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.330+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe7804eeb0 con 0x7fbe9013f710 2024-03-25T19:52:21.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.330+0000 7fbe86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:21.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.330+0000 7fbe86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:21.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.579+0000 7fbe94d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:21.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.579+0000 7fbe94d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:21.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.579+0000 7fbe94d4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T19:52:21.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.579+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe901d7930 con 0x7fbe9013f710 2024-03-25T19:52:21.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.631+0000 7fbe86ffd640 1 -- 172.21.15.18:0/540088892 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3260303895) 0x7fbe78050fb0 con 0x7fbe9013f710 2024-03-25T19:52:21.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.631+0000 7fbe86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:21.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.631+0000 7fbe86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:21.631 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T19:52:21.632 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"e9758276-eabb-4964-9819-31fbe65d0466","created":"2024-03-25T19:40:49.820738+0000","modified":"2024-03-25T19:50:53.820580+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:50:53.820580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"36fa8467-9d3a-44b4-8cc6-a8d49dd9e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8dacde5d-1a0c-4aca-a176-862be9c3a92f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4095cdd1-ecc3-47d3-9923-7444cab97271","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f3ca72b0-c0c2-4d82-a95d-814c63529db1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d48b076-42ea-4b65-81d9-8fe8cdc9b352","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5ca4d285-71b8-402d-b33c-496cbe7cf78f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"32f1487e-1fcc-417d-b7e3-e42a604b5451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c0d8be8d-8904-4537-9d20-c44e153b010b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.633+0000 7fbe94d4d640 10 librados: watch_flush enter 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.633+0000 7fbe94d4d640 10 librados: watch_flush exit 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.633+0000 7fbe94d4d640 20 client.4709.objecter shutdown clearing up homeless session... 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.633+0000 7fbe94d4d640 10 client.4709.objecter successfully canceled tick 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.633+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 >> v1:172.21.15.18:6800/1520000573 conn(0x7fbe7006de90 legacy=0x7fbe70070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.633+0000 7fbe94d4d640 10 monclient: shutdown 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.633+0000 7fbe94d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.633+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 >> v1:172.21.15.18:6790/0 conn(0x7fbe9013f710 legacy=0x7fbe901d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.634+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 shutdown_connections 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.634+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 >> 172.21.15.18:0/540088892 conn(0x7fbe900b7ea0 msgr2=0x7fbe90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:21.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.634+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 shutdown_connections 2024-03-25T19:52:21.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.634+0000 7fbe94d4d640 1 -- 172.21.15.18:0/540088892 wait complete. 2024-03-25T19:52:21.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T19:52:21.634+0000 7fbe94d4d640 1 librados: shutdown 2024-03-25T19:52:21.647 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:21.647 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:52:21.648 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:52:21.649 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:52:21.649 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:23.571 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:11:28.213 35800== 2024-03-25T19:52:23.571 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:11:28.213 35800== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:23.608 INFO:tasks.ceph.osd.0.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:24.277 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:51.807 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.294 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:51.824 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.327 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:51.857 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.331 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:51.827 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.339 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:51.835 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.340 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:51.871 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.350 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:51.881 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.371 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:11:51.867 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:24.384 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:51.915 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:27.751 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:52:27.752 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:52:27.752 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:52:27.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:29.646 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:11:34.270 35802== 2024-03-25T19:52:29.646 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:11:34.270 35802== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:29.682 INFO:tasks.ceph.osd.1.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:33.854 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:52:33.855 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:52:33.855 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:52:33.855 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:35.828 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:11:40.459 35801== 2024-03-25T19:52:35.828 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:11:40.459 35801== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:35.879 INFO:tasks.ceph.osd.2.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:39.957 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:52:39.958 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:52:39.958 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:52:39.958 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:41.704 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:11:46.327 35803== 2024-03-25T19:52:41.704 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:11:46.327 35803== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:41.753 INFO:tasks.ceph.osd.3.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:44.283 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:11.813 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:44.331 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:12:11.827 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:44.331 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:11.862 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:44.338 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:12:11.834 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:44.356 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:11.887 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:44.364 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:12:11.860 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:44.380 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:11.911 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:44.384 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:11.914 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:44.409 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:11.939 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:46.060 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:52:46.060 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:52:46.061 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:52:46.061 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:48.157 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:11:52.795 35215== 2024-03-25T19:52:48.157 INFO:tasks.ceph.osd.4.smithi053.stderr:==00:00:11:52.795 35215== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:48.196 INFO:tasks.ceph.osd.4.smithi053.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:52.163 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:52:52.164 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:52:52.164 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:52:52.164 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:53.973 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:11:58.587 35218== 2024-03-25T19:52:53.973 INFO:tasks.ceph.osd.5.smithi053.stderr:==00:00:11:58.587 35218== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:54.014 INFO:tasks.ceph.osd.5.smithi053.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:58.266 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:52:58.266 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:52:58.267 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:52:58.267 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:00.105 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:12:04.719 35217== 2024-03-25T19:53:00.106 INFO:tasks.ceph.osd.6.smithi053.stderr:==00:00:12:04.719 35217== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:00.145 INFO:tasks.ceph.osd.6.smithi053.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:04.369 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:53:04.369 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:53:04.370 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:53:04.370 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:04.380 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:31.911 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:04.383 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:31.913 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:04.394 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:31.925 35488== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:04.431 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:31.962 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:04.431 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:12:31.927 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:04.432 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:31.963 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:04.433 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:12:31.929 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:04.443 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:31.974 35489== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:04.444 INFO:tasks.ceph.mon.b.smithi053.stderr:==00:00:12:31.940 34671== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.005 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:12:10.631 35216== 2024-03-25T19:53:06.005 INFO:tasks.ceph.osd.7.smithi053.stderr:==00:00:12:10.631 35216== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:06.036 INFO:tasks.ceph.osd.7.smithi053.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:10.471 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:53:10.472 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:53:10.472 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:53:10.472 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:53:10.473 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:10.473 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T19:53:10.474+0000 7fc73ca77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35494) UID: 0 2024-03-25T19:53:10.500 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:53:10.500 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:53:10.500 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:10.528 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:53:10.528 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:53:10.528 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:53:10.528 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:10.558 INFO:tasks.ceph.mon.a.smithi018.stderr:2024-03-25T19:53:10.546+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35454) UID: 0 2024-03-25T19:53:10.559 INFO:tasks.ceph.mon.a.smithi018.stderr:2024-03-25T19:53:10.547+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:58:09.602 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:58:09.603 DEBUG:teuthology.orchestra.run.smithi018:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:58:09.650 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:58:09.650 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:58:09.865 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:58:09.865 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:58:09.931 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:58:09.931 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:58:09.984 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:58:09.984 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:58:10.057 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:58:10.058 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:58:10.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:58:10.156 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:58:10.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:58:10.220 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:58:10.279 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:58:10.279 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:58:10.353 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:58:10.354 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/data/mon.a.tgz 2024-03-25T19:58:10.355 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-25T19:58:10.369 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.ypZYh3m7uI 2024-03-25T19:58:10.369 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ypZYh3m7uI 2024-03-25T19:58:10.463 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.ypZYh3m7uI 2024-03-25T19:58:10.546 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.ypZYh3m7uI is 134KB 2024-03-25T19:58:10.598 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.ypZYh3m7uI 2024-03-25T19:58:10.611 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/data/mon.c.tgz 2024-03-25T19:58:10.611 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-25T19:58:10.665 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.9aFJNypIDy 2024-03-25T19:58:10.666 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.9aFJNypIDy 2024-03-25T19:58:10.757 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.9aFJNypIDy 2024-03-25T19:58:10.840 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.9aFJNypIDy is 165KB 2024-03-25T19:58:10.890 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.9aFJNypIDy 2024-03-25T19:58:10.903 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/data/mon.b.tgz 2024-03-25T19:58:10.904 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-03-25T19:58:10.921 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.5Khz7lCtbK 2024-03-25T19:58:10.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.5Khz7lCtbK 2024-03-25T19:58:11.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.5Khz7lCtbK 2024-03-25T19:58:11.120 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.5Khz7lCtbK is 156KB 2024-03-25T19:58:11.172 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.5Khz7lCtbK 2024-03-25T19:58:11.187 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:58:11.187 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:58:11.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:58:11.329 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:58:11.375 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:58:11.568 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:58:11.647 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:58:11.792 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:58:11.793 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/remote/smithi018/crash 2024-03-25T19:58:11.794 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:58:11.821 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/remote/smithi053/crash 2024-03-25T19:58:11.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:58:11.849 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:58:11.850 DEBUG:teuthology.orchestra.run.smithi018:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:58:11.865 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:58:11.939 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34985.log 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34825.log 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34865.log 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34985.log.gz 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34825.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34865.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.34825.log.gz 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose 87.0% -- /var/log/ceph/valgrind/osd.6.log -- replaced with /var/log/ceph/ceph-client.admin.34865.log.gz 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:58:11.940 INFO:teuthology.orchestra.run.smithi053.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35116.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr: 55.7% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.35116.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35065.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35116.log.gz 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.35065.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35065.log.gz 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34905.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.34905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34945.log 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34905.log.gz 2024-03-25T19:58:11.941 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:58:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34945.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:58:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34945.log.gz 2024-03-25T19:58:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35025.log 2024-03-25T19:58:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.35025.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:58:11.942 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35025.log.gz 2024-03-25T19:58:11.942 INFO:teuthology.orchestra.run.smithi053.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40305.log 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42703.log 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41281.log 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40305.log.gz 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41281.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42703.log.gz 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36052.log 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41281.log.gz 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42352.log 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-03-25T19:58:11.947 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.36052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39525.log 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40383.log 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36052.log.gz 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42352.log.gz 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40032.log 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41710.log 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40383.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39525.log.gz 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40032.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29645.log 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40383.log.gz 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41710.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40032.log.gz 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39915.log 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39759.log 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph.tmp-client.admin.29645.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29645.log.gz 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41710.log.gz 2024-03-25T19:58:11.948 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39564.log 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40149.log 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39759.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39915.log.gz 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39759.log.gz 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40149.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39564.log.gzgzip 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41515.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40149.log.gz 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr: 55.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:58:11.949 INFO:teuthology.orchestra.run.smithi018.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/osd.1.log: /var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36916.log 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr: 55.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42079.log 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.36916.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41007.log 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose 87.0% -- -- replaced with /var/log/ceph/ceph-client.admin.36916.log.gz /var/log/ceph/ceph-client.admin.41749.log 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42079.log: /var/log/ceph/ceph-client.admin.41007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39486.log 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41749.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42079.log.gz 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39876.log 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41007.log.gz 2024-03-25T19:58:11.950 INFO:teuthology.orchestra.run.smithi018.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41749.log.gz 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40266.log 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40578.log 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3%/var/log/ceph/ceph-client.admin.40266.log: -- replaced with /var/log/ceph/ceph-client.admin.39486.log.gz 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41359.log 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39876.log.gz 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41398.log 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40266.log.gz 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40422.log 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40578.log.gz 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39720.log 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41359.log.gz 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40734.log 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41398.log.gz 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39720.log: /var/log/ceph/ceph-client.admin.40929.log 2024-03-25T19:58:11.951 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40422.log.gz 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40071.log 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39720.log.gz 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42586.log 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40734.log.gz 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40500.log 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42586.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40071.log.gz 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41671.log 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40500.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42586.log.gz 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42469.log 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40500.log.gz 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph.audit.log: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.41671.log.gz 2024-03-25T19:58:11.952 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40773.log 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41086.log 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42469.log: /var/log/ceph/ceph-client.admin.40773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39837.log 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42469.log.gz 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42664.log 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40773.log.gz 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40227.log 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39837.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41086.log.gz 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41476.log 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39837.log.gz 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42118.log 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42664.log.gz 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40227.log.gz 2024-03-25T19:58:11.953 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42118.log: gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41476.log.gz 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41788.log 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42118.log.gz 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42196.log 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41164.log 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39170.log 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41788.log.gz 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41866.log 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42196.log.gz 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39170.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41164.log.gz 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41866.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39170.log.gz 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39603.log 2024-03-25T19:58:11.954 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42547.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.1.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41866.log.gz 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40968.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39603.log.gz 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41593.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42547.log.gz 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40851.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41242.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40968.log.gz 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40851.log: gzip -5 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41593.log.gz --verbose 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr: -- /var/log/ceph/ceph-client.admin.35904.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39798.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40851.log.gz 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.35904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39954.log 2024-03-25T19:58:11.955 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41242.log.gz 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39798.log: gzip -5 --verbose -- 85.9% -- replaced with /var/log/ceph/ceph-client.admin.35904.log.gz 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr: /var/log/ceph/ceph-client.admin.41944.log 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39954.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39798.log.gz 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39681.log 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41944.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39954.log.gz 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41827.log 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41944.log.gz 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39681.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41047.log 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41827.log: /var/log/ceph/ceph-osd.3.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39681.log.gz 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40188.log 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41047.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41827.log.gz 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35496.log 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40188.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41047.log.gz 2024-03-25T19:58:11.956 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42274.log 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.35496.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40188.log.gz 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41905.log 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35799.log 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35496.log.gz 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42274.log.gz 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40539.log 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9%/var/log/ceph/ceph-client.admin.35799.log: -- replaced with /var/log/ceph/ceph-client.admin.41905.log.gz 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- 85.7% /var/log/ceph/ceph-client.admin.40812.log 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35799.log.gz 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40539.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40812.log: gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40539.log.gz 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40656.log 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-mgr.y.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2024-03-25T19:58:11.957 INFO:teuthology.orchestra.run.smithi018.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42038.log 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2%/var/log/ceph/ceph-client.admin.40344.log: -- replaced with /var/log/ceph/ceph-client.admin.40656.log.gz 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42038.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42038.log.gz 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42157.log 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:gzip/var/log/ceph/ceph-osd.0.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39445.log 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41437.log 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39445.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42157.log.gz 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39445.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41985.log 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40461.log 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41437.log.gz 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41985.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41985.log.gz 2024-03-25T19:58:11.958 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42391.log 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40461.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40461.log.gz 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42508.log 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42430.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42391.log.gz 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.42508.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42508.log.gz 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42235.log 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.2.log: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39642.log 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42235.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42235.log.gz 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39993.log 2024-03-25T19:58:11.959 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39642.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39642.log.gz 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41203.log 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39993.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39993.log.gz 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37890.log 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41203.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41203.log.gz 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42625.log 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.37890.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37890.log.gz 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41320.log 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42625.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42625.log.gz 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40617.log 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41320.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41320.log.gz 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40110.log 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40617.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40617.log.gz 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40110.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40110.log.gz 2024-03-25T19:58:11.960 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42313.log 2024-03-25T19:58:11.961 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39406.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-03-25T19:58:11.961 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-03-25T19:58:11.961 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42313.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42313.log.gz 2024-03-25T19:58:11.961 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41554.log 2024-03-25T19:58:11.961 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41125.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-03-25T19:58:11.961 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:58:11.961 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41554.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41554.log.gz 2024-03-25T19:58:11.993 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-mon.c.log: 92.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:58:12.695 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:17:40.225 35488== 2024-03-25T19:58:12.695 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:17:40.225 35488== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:58:12.712 INFO:tasks.ceph.mon.a.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:58:12.836 INFO:teuthology.orchestra.run.smithi053.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:58:12.847 INFO:teuthology.orchestra.run.smithi053.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:58:13.110 INFO:teuthology.orchestra.run.smithi053.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:58:13.116 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:58:13.118 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:58:13.137 INFO:teuthology.orchestra.run.smithi053.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:58:13.147 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:58:13.152 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:58:16.827 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2024-03-25T19:58:16.827 INFO:teuthology.orchestra.run.smithi018.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:58:16.981 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2024-03-25T19:58:17.044 INFO:teuthology.orchestra.run.smithi018.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:58:17.046 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T19:58:17.046 INFO:teuthology.orchestra.run.smithi018.stderr:real 0m5.169s 2024-03-25T19:58:17.046 INFO:teuthology.orchestra.run.smithi018.stderr:user 0m14.615s 2024-03-25T19:58:17.046 INFO:teuthology.orchestra.run.smithi018.stderr:sys 0m0.493s 2024-03-25T19:58:17.069 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2024-03-25T19:58:17.069 INFO:teuthology.orchestra.run.smithi053.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:58:17.070 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-25T19:58:17.071 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m5.161s 2024-03-25T19:58:17.071 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m9.156s 2024-03-25T19:58:17.071 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.256s 2024-03-25T19:58:17.071 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:58:17.071 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/remote/smithi018/log 2024-03-25T19:58:17.072 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:58:19.440 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/remote/smithi053/log 2024-03-25T19:58:19.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:58:20.998 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:58:21.215 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cd327b9a3e1a4c21b4150bcf56604520 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:58:21.221 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:58:21.230 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:58:21.239 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:58:21.239 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:58:21.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:58:21.337 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:58:21.347 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:58:21.347 DEBUG:teuthology.orchestra.run.smithi018:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:58:21.350 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:58:21.364 INFO:teuthology.orchestra.run.smithi018.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:58:21.366 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:58:21.369 INFO:teuthology.orchestra.run.smithi018.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:58:21.369 INFO:teuthology.orchestra.run.smithi018.stdout:=============================================================================== 2024-03-25T19:58:21.369 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 -534us[ -534us] +/- 67ms 2024-03-25T19:58:21.369 INFO:teuthology.orchestra.run.smithi018.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -125us[ -115us] +/- 69ms 2024-03-25T19:58:21.369 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 +782us[ +792us] +/- 72ms 2024-03-25T19:58:21.370 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:58:21.370 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:58:21.370 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-03-25T19:58:21.370 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 -528us[ -486us] +/- 67ms 2024-03-25T19:58:21.370 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -99us[ -56us] +/- 69ms 2024-03-25T19:58:21.371 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +850us[ +850us] +/- 73ms 2024-03-25T19:58:21.371 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:58:21.372 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:58:21.393 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:58:21.394 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:58:21.408 DEBUG:teuthology.orchestra.run.smithi018:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:58:21.480 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:58:21.553 DEBUG:teuthology.orchestra.run.smithi018:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:58:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:21.585 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:58:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:21.616 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:58:21.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:58:21.642 INFO:teuthology.task.internal:Duration was 5573.476065 seconds 2024-03-25T19:58:21.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:58:21.658 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:58:21.658 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:58:21.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:58:21.703 INFO:teuthology.orchestra.run.smithi018.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:58:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:58:21.980 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:58:21.980 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T19:58:21.980 DEBUG:teuthology.orchestra.run.smithi018:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:58:22.000 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-03-25T19:58:22.001 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:58:22.022 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:58:22.023 DEBUG:teuthology.orchestra.run.smithi018:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:58:22.042 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:58:22.091 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:58:22.092 DEBUG:teuthology.orchestra.run.smithi018:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:58:22.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:58:22.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:58:22.247 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:58:22.247 DEBUG:teuthology.orchestra.run.smithi018:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:58:22.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:58:22.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:58:22.314 DEBUG:teuthology.orchestra.run.smithi018:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:22.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:22.339 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = core 2024-03-25T19:58:22.367 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-03-25T19:58:22.393 DEBUG:teuthology.orchestra.run.smithi018:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:22.416 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:22.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:58:22.457 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:58:22.457 DEBUG:teuthology.misc:Transferring archived files from smithi018:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/remote/smithi018 2024-03-25T19:58:22.458 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:58:22.494 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323/remote/smithi053 2024-03-25T19:58:22.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:58:22.531 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:58:22.531 DEBUG:teuthology.orchestra.run.smithi018:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:58:22.533 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:58:22.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:58:22.595 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:58:22.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:58:22.655 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:58:22.655 DEBUG:teuthology.orchestra.run.smithi018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:58:22.658 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:58:22.670 INFO:teuthology.orchestra.run.smithi018.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:58 /home/ubuntu/cephtest 2024-03-25T19:58:22.671 INFO:teuthology.orchestra.run.smithi053.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:58 /home/ubuntu/cephtest 2024-03-25T19:58:22.673 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:58:22.685 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:58:22.717 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:58:22.743 DEBUG:teuthology.nuke:shortname: smithi018 2024-03-25T19:58:22.745 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:58:22.770 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:10.922456', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIf8qtBW1VRnU77oStvxOR7mFLCKGb1HTnzzU0QgDPNLlMjYNsWy8DxR7LMFeOQOQ9GDGrEIsPPh/wjTS8d9ASk='} 2024-03-25T19:58:22.771 DEBUG:teuthology.nuke:shortname: smithi053 2024-03-25T19:58:22.771 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:58:22.796 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621323', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:10.924560', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4uEslQyHioVMal//nl/QuVFFOzb5Vxx/XC+yGG8DaL9Py/goUzcz2fw0wljTg06zHW5budGXLp7J9jX4RObhg='} 2024-03-25T19:58:22.819 INFO:teuthology.orchestra.console.smithi018:Power off 2024-03-25T19:58:22.820 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:58:22.850 DEBUG:teuthology.orchestra.console.smithi018:power off output: Chassis Power Control: Down/Off 2024-03-25T19:58:22.851 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:22.867 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-25T19:58:22.867 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:58:22.878 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is on 2024-03-25T19:58:22.893 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-25T19:58:22.893 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:22.917 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-25T19:58:26.879 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:26.918 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:26.989 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is on 2024-03-25T19:58:27.031 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-25T19:58:30.994 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:31.032 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:31.106 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is off 2024-03-25T19:58:31.143 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-25T19:58:31.207 INFO:teuthology.orchestra.console.smithi018:Power off completed 2024-03-25T19:58:31.244 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-25T19:58:31.349 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 5573.476064682007 failure_reason: 'valgrind error: Leak_PossiblyLost calloc __trans_list_add ' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cd327b9a3e1a4c21b4150bcf56604520 status: fail success: false 2024-03-25T19:58:31.349 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:58:31.445 INFO:teuthology.run:FAIL